DevOps-jenkins-SonarQube-1

架构(代码质量平台集成-SonarQube)

一台 SonarQube Server启动3个主要过程：

	1.  Web服务器，供开发人员，管理人员浏览高质量的快照并配置SonarQube实例
	2.  基于Elastiesearch的Search Server从UI进行后退搜索
	3.  Compute Engine服务器，负责处理代码分析报告并将其保存在SonarQube数据库中

一个SonarQube数据库要存储：

	1.  SonarQube实例的配置（安全性，插件设置等）
	2.  项目，视图等的质量快照。
	3. 服务器上安装了多个SonarQube插件，可能包括语言，SCM，集成，身份验证和管理插件
	4.  在构建/持续集成服务器上运行一个或多个SonarScanner，以分析项目

工作原理

1. 开发人员在他们的 IDE 中编码并使用 SonarLint 运行本地分析。
2. 开发人员把代码推送到 Git 等代码管理仓库
3. 通过 Jenkens 等持续集成服务器，并使用 sonar-scanner 进行代码分析
4. 分析报告被发送到 SonarQube 服务器进行处理。
5. SonarQube Server 将分析报告结果处理并存储在 SonarQube 数据库中，并在 UI 中显示结果。
6. 开发人员通过 SonarQube UI 审查、评论、挑战他们的问题，以管理和减少他们的技术债务。
7. 使用 API 来自动化配置并从 SonarQube 中提取数据。

CentOS7部署SonarQube

下载链接： https://www.sonarqube.org/downloads/

1. 创建用户
[root@worker-2 ~]# useradd sonarqube
[root@worker-2 ~]# passwd sonarqube

2. 优化系统参数
[root@worker-2 ~]# sysctl -w  vm.max_map_count=262144
[root@worker-2 ~]# sysctl -w fs.file-max=65536
[root@worker-2 ~]# ulimit -u 4096 sonarqube
[root@worker-2 ~]# ulimit -n 65536 sonarqube

3. 解压文件,需要在sonarqube的家目录下执行
[root@worker-2 ~]# mv sonarqube-8.9.9.56886.zip /home/sonarqube/
[root@worker-2 ~]# cd /home/sonarqube/
[root@worker-2 sonarqube]# ll
total 258052
-rw-r--r-- 1 root root 264242675 Jul 22 09:41 sonarqube-8.9.9.56886.zip
[root@worker-2 sonarqube]# chown -R sonarqube:sonarqube /home/sonarqube/
[root@worker-2 sonarqube]# su - sonarqube
[sonarqube@worker-2 ~]$ unzip sonarqube-8.9.9.56886.zip 

4. 更改配置文件
[sonarqube@worker-2 ~]$ grep -v "^#" sonarqube-8.9.9.56886/conf/sonar.properties |grep -v "^$"
sonar.jdbc.username=sonarqube
sonar.jdbc.password=li****4.
sonar.jdbc.url=jdbc:postgresql://43.142.117.79/sonarqube

5. 安装 PostgreSQL
[root@worker-2 ~]# wget https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
[root@worker-2 ~]# yum install -y pgdg-redhat-repo-latest.noarch.rpm
[root@worker-2 ~]# yum install -y postgresql10-server postgresql10
# 初始化
[root@worker-2 ~]# /usr/pgsql-10/bin/postgresql-10-setup initdb
Initializing database ... OK
[root@worker-2 ~]# systemctl start postgresql-10
# 启动、自启
[root@worker-2 ~]# systemctl start  postgresql.service 
[root@worker-2 ~]# systemctl  enable postgresql.service 
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql.service to /usr/lib/systemd/system/postgresql.service.
[root@worker-2 ~]# firewall-cmd --add-service=postgresql --permanent
FirewallD is not running
[root@worker-2 ~]# firewall-cmd --reload
FirewallD is not running
# 登入数据库配置
[root@worker-2 ~]# su - postgres
Last failed login: Mon Jul 18 22:23:31 CST 2022 from 119.108.220.121 on ssh:notty
There were 4 failed login attempts since the last successful login.
# 创建用户并设置密码
-bash-4.2$ psql
psql (9.2.24)
Type "help" for help.
postgres=# create user sonar with password 'lippdpp4.';
postgres=# create database sonarqube owner sonar;

postgres=# ALTER USER postgres with encrypted password 'lippdpp4.';    # 修改密码方式
ALTER ROLE
postgres=# \q
-bash-4.2$ exit
logout

# 开启远程访问
[root@worker-2 ~]# vim /var/lib/pgsql/10/data/postgresql.conf
listen_addresses = '*'     # 设置监听地址为*
[root@worker-2 ~]# vim /var/lib/pgsql/10/data/pg_hba.conf
# 在 IPv4 local connections  下一行添加
# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            ident
host    all             all             0.0.0.0/0            trust
# IPv6 local connections:
host    all             all             ::1/128                 ident
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            ident
host    replication     all             ::1/128                 ident
# 重启服务
[root@worker-2 ~]# systemctl restart postgresql-10

6. 添加系统服务
[root@worker-2 ~]# ln -s /home/sonarqube/sonarqube-8.9.9.56886/bin/linux-x86-64/sonar.sh /usr/bin/sonar
# 创建自启文件
[root@worker-2 ~]# vim /etc/init.d/sonarqube
#!/bin/sh
#
# rc file for SonarQube
#
# chkconfig: 345 96 10
# description: SonarQube system (www.sonarsource.org)
#
### BEGIN INIT INFO
# Provides: sonar
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: SonarQube system (www.sonarsource.org)
# Description: SonarQube system (www.sonarsource.org)
### END INIT INFO
 
/usr/bin/sonar $*

[root@worker-2 ~]# chkconfig --add sonarqube
[root@worker-2 ~]# chkconfig sonarqube on

# 授权并启动
[root@worker-2 ~]# chmod 777  /etc/init.d/sonarqube
[root@worker-2 ~]# /etc/init.d/sonarqube start
Starting SonarQube...
Started SonarQube.
[root@worker-2 ~]# chown -R sonarqube:sonarqube /home/sonarqube/
[root@worker-2 ~]# service sonarqube status
SonarQube is running (5129).

测试连接

遇到的报错

1.java.lang.IllegalArgumentException: Unable to create shared memory :

/home/sonarqube/sonarqube-8.9.9.56886/temp。权限不足
[root@worker-2 ~]# chown -R sonarqube:sonarqube /home/sonarqube/

2. 优化系统参数未优化
   
   优化系统参数未优化

sysctl -w  vm.max_map_count=262144
sysctl -w fs.file-max=65536
ulimit -u 4096 sonarqube
ulimit -n 65536 sonarqube

3. 上次运行的程序没有关闭
   
   测试
   

SonarQube 的使用

登入并修改admin账户密码

配置-修改中文界面(确定安装三方插件后重启服务生效)

配置-集成GitLab

生成app令牌

sonarqube配置


测试

安装配置sonarscanner(可以在任意机器)

官方文档 https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/

安装

下载SonarScanner(linux 64-bit) 安装文件并安装
unzip sonar-scanner-cli-4.7.0.2747-linux.zip -d /usr/local
vim /etc/profile
export SCANNER_HOME=/usr/local/sonar-scanner-4.7.0.2747-linux
export PATH=$PATH:$GROOVY_HOME/bin:$MAVEN_HOME/bin:$ANT_HOME/bin:$GRADLE_HOME/bin:$NODE_HOME/bin:$SCANNER_HOME/bin
source /etc/profile

配置

[root@worker-1 ~]# vim /usr/local/sonar-scanner-4.7.0.2747-linux/conf/sonar-scanner.properties 
sonar.host.url=http://43*****:9000     # 指定sonarqube所在url
sonar.sourceEncoding=UTF-8
sonar.login=admin                      # 指定sonarqube的账户密码
sonar.password=l***p4.
sonar.java.binaries=target/classes     # 高版本必须指定java   类,扫描前先 mvn clean package

测试-在项目文件中创建shell 文件,或者在项目文件目录运行命令

vim start.sh
#!/bin/bash

projectName="demo-maven-service"
scanTime=`date +%Y%m%d%H%M%S`

for((i=1;i<=10;i++))
do
    sonar-scanner  -Dsonar.host.url=http://43.142.117.79:9000  \
    -Dsonar.projectKey=${projectName}${i}  \
    -Dsonar.projectName=${projectName}${i}  \
    -Dsonar.projectVersion=${scanTime} \
# 参数写入了sonar-scanner 中的config 
#    -Dsonar.login=admin \
#    -Dsonar.password=lippdpp4. \
    -Dsonar.projectDescription="my first project!"  \
    -Dsonar.links.homepage=http://www.baidu.com \
    -Dsonar.sources=src \
    -Dsonar.sourceEncoding=UTF-8 \
    -Dsonar.java.binaries=target/classes \
    -Dsonar.java.test.binaries=target/test-classes \
    -Dsonar.java.surefire.report=target/surefire-reports

    echo "${projectName}  scan success!"
done

[root@worker-1 demo-maven-service]# sh start.sh 
INFO: Scanner configuration file: /usr/local/sonar-scanner-4.7.0.2747-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: /root/k8s/gitlab/demo-maven-service/sonar-project.properties
INFO: SonarScanner 4.7.0.2747
INFO: Java 11.0.14.1 Eclipse Adoptium (64-bit)
INFO: Linux 3.10.0-1160.62.1.el7.x86_64 amd64
INFO: User cache: /root/.sonar/cache
INFO: Scanner configuration file: /usr/local/sonar-scanner-4.7.0.2747-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: /root/k8s/gitlab/demo-maven-service/sonar-project.properties
INFO: Analyzing on SonarQube server 8.9.9
INFO: Default locale: "en_US", source code encoding: "UTF-8"
INFO: Load global settings
INFO: Load global settings (done) | time=223ms
INFO: Server id: B7208F76-AYIpktbZvLnuw12ZbKf6
INFO: User cache: /root/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=170ms
INFO: Plugin [l10nzh] defines 'l10nen' as base plugin. This metadata can be removed from manifest of l10n plugins since version 5.2.
INFO: Load/download plugins (done) | time=230ms
INFO: Process project properties
INFO: Process project properties (done) | time=8ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=1ms
INFO: Project key: demo-maven-service1
INFO: Base dir: /root/k8s/gitlab/demo-maven-service
INFO: Working dir: /root/k8s/gitlab/demo-maven-service/.scannerwork
INFO: Load project settings for component key: 'demo-maven-service1'
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=411ms
INFO: Load active rules
INFO: Load active rules (done) | time=10913ms
INFO: Indexing files...
INFO: Project configuration:
INFO: 20 files indexed
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for java: Sonar way
INFO: Quality profile for xml: Sonar way
INFO: ------------- Run sensors on module demo-maven-service1
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=177ms
INFO: Sensor JavaSquidSensor [java]
INFO: Configured Java source version (sonar.java.source): none
INFO: JavaClasspath initialization
INFO: JavaClasspath initialization (done) | time=8ms
INFO: JavaTestClasspath initialization
INFO: JavaTestClasspath initialization (done) | time=0ms
INFO: Java Main Files AST scan
INFO: 2 source files to be analyzed
INFO: Load project repositories
INFO: Load project repositories (done) | time=136ms
INFO: 2/2 source files have been analyzed
WARN: Dependencies/libraries were not provided for analysis of SOURCE files. The 'sonar.java.libraries' property is empty. Verify your configuration, as you might end up with less precise results.
WARN: Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
INFO: Java Main Files AST scan (done) | time=1577ms
INFO: Java Test Files AST scan
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Java Test Files AST scan (done) | time=9ms
INFO: Java Generated Files AST scan
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Java Generated Files AST scan (done) | time=0ms
INFO: Sensor JavaSquidSensor [java] (done) | time=1978ms
INFO: Sensor CSS Rules [cssfamily]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [cssfamily] (done) | time=0ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=4ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=1ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=1ms
INFO: Sensor SurefireSensor [java]
INFO: parsing [/root/k8s/gitlab/demo-maven-service/target/surefire-reports]
INFO: Sensor SurefireSensor [java] (done) | time=53ms
INFO: Sensor JavaXmlSensor [java]
INFO: 2 source files to be analyzed
INFO: 2/2 source files have been analyzed
INFO: Sensor JavaXmlSensor [java] (done) | time=162ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=3ms
INFO: Sensor XML Sensor [xml]
INFO: 2 source files to be analyzed
INFO: 2/2 source files have been analyzed
INFO: Sensor XML Sensor [xml] (done) | time=188ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=1ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=1ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=7ms
INFO: Sensor Java CPD Block Indexer
INFO: Sensor Java CPD Block Indexer (done) | time=25ms
INFO: SCM Publisher SCM provider for this project is: git
INFO: SCM Publisher 4 source files to be analyzed
INFO: SCM Publisher 3/4 source files have been analyzed (done) | time=166ms
WARN: Missing blame information for the following files:
WARN:   * target/surefire-reports/TEST-com.mycompany.app.AppTest.xml
WARN: This may lead to missing/broken features in SonarQube
INFO: CPD Executor 1 file had no CPD blocks
INFO: CPD Executor Calculating CPD for 1 file
INFO: CPD Executor CPD calculation finished (done) | time=8ms
INFO: Analysis report generated in 158ms, dir size=110 KB
INFO: Analysis report compressed in 17ms, zip size=21 KB
INFO: Analysis report uploaded in 714ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://URL:9000/dashboard?id=demo-maven-service1
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://URL:9000/api/ce/task?id=AYIqNb3UbF7mLcOMVWBD
INFO: Analysis total time: 17.004 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 18.384s
INFO: Final Memory: 9M/37M
INFO: ------------------------------------------------------------------------
start.sh: line 15: -Dsonar.ws.timeout=30: command not found
demo-maven-service  scan success!

遇到的报错

1. 高版本需要制定java 类, 增加 sonar.java.binaries=target/classes
   
2. sonarqube 的账户信息错误

SonarQube 集成到 Jenkins

Jenkins 配置

Jenkinslib 共享库 jenkinslib/src/org/devops/sonarqube.groovy

package org.devops

//scan
def SonarScan(projectName,projectDesc,projectPath){
    def scannerHome = "/usr/local/sonar-scanner-4.7.0.2747-linux"
    def sonarServer = "http://43.****79:9000"
    scanDate = sh returnStdout: true, script: 'date +%Y%m%d%H%M%S'
    scanDate = scanDate - "\n"
    sh """
        #!/bin/bash

        ${scannerHome}/bin/sonar-scanner  -Dsonar.host.url=${sonarServer}  \
                -Dsonar.projectKey=${projectName} \
                -Dsonar.projectName=${projectName} \
                -Dsonar.projectVersion=${scanDate} \
                -Dsonar.login=admin \
                -Dsonar.password=li****4. \
                -Dsonar.projectDescription=${projectDesc}  \
                -Dsonar.links.homepage=http://www.baidu.com \
                -Dsonar.sources=${projectPath} \
                -Dsonar.sourceEncoding=UTF-8 \
                -Dsonar.java.binaries=target/classes \
                -Dsonar.java.test.binaries=target/test-classes \
                -Dsonar.java.surefire.report=target/surefire-reports

                echo "${projectName}  scan success!"
        
    """
}

Jenkinsfile

def sonar = new org.devops.sonarqube()

...
        stage("QA"){
            steps{
                script{
                    tools.PrintMes("代码扫描","green")
                    sonar.SonarScan("${JOB_NAME}","${JOB_NAME}","src")   //传入函数
                }
            }
        }

测试


可以看到扫描的版本

查看存在的问题

SonarQube 优化

JenkinsSonar 插件

安装插件

Sonar 生成另外并添加到 jenkins 凭证

配置Jenkins

修改 sonarqube.groovy

package org.devops

//scan
def SonarScan(projectName,projectDesc,projectPath){
	//指定了sonar集群(jenkins 配置的),就可以省略 sonar-url和账户及密码信息
    withSonarQubeEnv('sonarqube-admin'){
        def scannerHome = "/usr/local/sonar-scanner-4.7.0.2747-linux"

        scanDate = sh returnStdout: true, script: 'date +%Y%m%d%H%M%S'
        scanDate = scanDate - "\n"
        sh """

            #!/bin/bash
            ${scannerHome}/bin/sonar-scanner  -Dsonar.projectKey=${projectName} \
                    -Dsonar.projectName=${projectName} \
                    -Dsonar.projectVersion=${scanDate} \
                    -Dsonar.projectDescription=${projectDesc}  \
                    -Dsonar.links.homepage=http://www.baidu.com \
                    -Dsonar.login=admin \
                    -Dsonar.password=lippdpp4. \
                    -Dsonar.sources=${projectPath} \
                    -Dsonar.sourceEncoding=UTF-8 \
                    -Dsonar.java.binaries=target/classes \
                    -Dsonar.java.test.binaries=target/test-classes \
                    -Dsonar.java.surefire.report=target/surefire-reports
                    echo "${projectName}  scan success!"
        """
    }
}

成效

有要求需要分测试和生产环境代码扫码

修改 Jenkinslib/jenkinslib/src/org/devops/sonarqube.groovy

package org.devops

//scan
def SonarScan(sonarServer,projectName,projectDesc,projectPath){
	//定义了一个servers函数,值就是jenkins设置的sonar名
	//SonarScan()函数也增加了sonarServer参数,需要在jenkinsfile文件中添加引用
    def servers = ["test":"sonarqube-admin","prod":"sonarqube-prod"]
    withSonarQubeEnv("${servers[sonarServer]}"){
        def scannerHome = "/usr/local/sonar-scanner-4.7.0.2747-linux"

        scanDate = sh returnStdout: true, script: 'date +%Y%m%d%H%M%S'
        scanDate = scanDate - "\n"

等等等


		stage("QA"){
            steps{
                script{
                    tools.PrintMes("代码扫描","green")
                    //添加参数引用test对应的值 sonarqube-admin
                    sonar.SonarScan("test","${JOB_NAME}","${JOB_NAME}","src")
                }
            }
        }