02 Powerful Plugin Functions of Jenkins

02Powerful Plugin Functions of Jenkins #

In the previous section, a brief introduction was given on the installation of Jenkins and some configuration instructions related to Jenkins system management. It was mentioned that the powerful functionality of Jenkins relies on the support of various plugins. Therefore, in this section, we will first introduce some common plugins and their related configuration instructions that will be used in this tutorial series.

After installing Jenkins, start the configuration by installing plugins. The following common plugins will be used in this tutorial:

  • Maven Integration plugin: Plugin for creating Maven projects.
  • Pipeline: Plugin for creating pipeline projects.
  • Ansible Plugin: Plugin for using Ansible.
  • Git/Git Parameter: Plugin for pulling Git code.
  • Docker Pipeline: Plugin for using Docker related syntax in Pipelines.
  • SonarQube Scanner: Plugin for executing SonarQube scans.
  • Publish Over SSH: Plugin for sending files to remote servers via SSH.
  • Email Extension Template: Plugin for sending emails to specific individuals or groups.
  • Role-based Authorization Strategy: Plugin for user authorization management.

Go back to the homepage, click “Manage Jenkins” -> “Manage Plugins” menu. Find the plugins and select the ones mentioned above, then click “Install” to install them.

For example, to install the Ansible plugin:

In the search box, enter the keyword and the plugins containing the keyword will be listed. Since I have already installed the Ansible plugin, it will not be listed here.

Configure Global Tools #

JDK #

Go back to the home page and click on “manage Jenkins -> Global Tool Configuration” menu.

First, you will see the configuration for the JDK tool. It has already been set up in the previous section, so you can simply import the configuration here. See the screenshot below.

Maven #

Scrolling down further, you will find the configuration for Maven tool. Since it hasn’t been installed yet, let’s configure it first.

Download it from the Apache official website (choose the version you need).

$ wget https://downloads.apache.org/maven/maven-3/3.6.3/source/apache-maven-3.5.4-src.tar.gz

After extraction, put it directly into the specified directory, and then add the environment variable configuration in /etc/profile. For example, here is my configuration.

$ tar xf apache-maven-3.5.4-src.tar.gz
$ vi /etc/profile 
export M2_HOME=/opt/apache-maven-3.5.4
export PATH=$PATH:$M2_HOME/bin

$ source /etc/profile

Verification.

$ mvn -version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: /opt/apache-maven-3.5.4
Java version: 1.8.0_231, vendor: Oracle Corporation, runtime: /usr/local/jdk1.8.0_231/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1062.4.1.el7.x86_64", arch: "amd64", family: "unix"

Then add the configuration information for Maven tool in Jenkins.

Note:

When Maven compiles the code, the settings.xml file in the default ${Maven_HOME}/conf/ path is configured to read the repository address (default is Aliyun repository) where common dependent JAR files are stored, as well as the username and password. Therefore, if you are using a different repository (such as a private Nexus repository), you need to modify this configuration file.

This is the completion of the Maven integration configuration in Jenkins, which is relatively simple.

Email Extension Template Plugin #

The Email Extension Template plugin is used to send emails to the users or groups to notify them when a job is executed.

Configure the Jenkins Location and Extended E-mail Notification, making sure that the system administrator’s email address matches the User Name value. See the example below.

-

Configure Default Email Subject and Content #

The parameters “Default Subject” and “Default Content” in this plugin are used to set the subject and content of the email to be sent. The default configuration is as follows:

Both of these parameters use the built-in global variables of Jenkins. $PROJECT_NAME represents the name of the project, $BUILD_NUMBER represents the ID of the project build, and $BUILD_STATUS represents the status of the project build. If you want to customize the subject and content of the email, you can modify them directly here.

After configuring, click “Test configuration” to test it.-

User Permission Control #

When using Jenkins, it is often necessary to assign different permissions to different users. Jenkins’s permission management is done through the “Role-based Authorization Strategy” plugin. The installation of this plugin is not covered in this demonstration. Now let’s take a look at how to enable this plugin.

Click “Manage Jenkins” -> “Configure Global Security”, and in the “Access Control” section, select “Role-Based Strategy” and save. After saving and exiting, you will see the “Manage and Assign Roles” option in “Manage Jenkins”, which is used to manage and configure user permissions.

Under the “Manage and Assign Roles” option, there are three menus, as shown below.

Roles

The usage process of this plugin is roughly as follows.

  • Use the “Manage Roles” option to create roles, which can be global roles or project roles. If it is a project role, you can create matching rules for the corresponding project role using regular expressions, so that this role has management permissions for certain projects.

  • Use the “Assign Roles” option to assign roles. The subject of role assignment is the users in the Jenkins system, which can be viewed through the “Manage Users” menu.

Creating Roles #

Click on Manage Roles, and after clicking inside, you will find a default admin role, which is a user role with administrator permissions.

Create a global role and a project role, as shown below.

In the Role to add input box, enter the role name (customized), and then enter the project name to be managed or the regular expression syntax to match multiple Jenkins projects (jobs) in the Pattern field. For example, I wrote test-.* here to match all projects starting with test-. After adding, click on the “Pattern” box to show the project names matched by this regular expression. Then, check the permissions for that role in the corresponding permission configuration column, such as what permissions there are for credentials, what permissions there are for jobs, etc. Here, you need to select according to your actual situation.

Role Allocation #

After creating roles, you can allocate roles to users. Who should the roles be allocated to? This depends on the users in Jenkins. Click “Manage-Jenkins” -> “Manage Users” to see all the users in the Jenkins system. Then go back to the “Manage and Assign Roles” menu and click “Assign Roles”. In the redirected interface, you can see the roles that were created just now. Similarly, you can assign both global roles and project roles to users.

As shown in the following figure.

Then you can log in using the test-li user. Note that this user needs to exist in Jenkins.

If you log in using the “test-li” account, you will find that there are no permissions, and the following error message is displayed.

This is because when creating the roles, a global role that has the permission to view the Jenkins dashboard was not created for the user. So at this point, you need to go to the “Manage Roles” menu to add a role with global permissions, as shown below:

Note:

Create a global role named test-all. The permission of this role is Overall/read, which means it has read permission for the Jenkins dashboard. The credentials column and agent column are not selected, indicating that it does not have permissions for Jenkins system management, credentials, node agents, etc.

The test-all role has read permission for the Jenkins dashboard, and after saving, you can reallocate permissions as shown below:

Allocate the test-li user with the permission to read the Jenkins dashboard, so that this user can work normally. As shown below:

That’s all for the explanation of role allocation for user permissions. This is roughly the process for creating roles and assigning permissions.

Sonar #

After configuring common plugins, let’s introduce the commonly used code quality inspection plugin in the continuous delivery process: Sonar.

Introduction #

Sonar is an open-source platform for code quality management. It can manage and detect the quality of source code from multiple dimensions. Through plugins, it supports code quality management and detection for over 20 programming languages, including Java, C#, C/C++, PL/SQL, Cobol, JavaScript, Groovy, etc. Sonar is also a popular code quality analysis tool used by many companies.

Sonar detects code quality from the following dimensions:

Sonar enforces code writing conventions through tools like PMD, CheckStyle, Findbugs, etc.

Sonar can also detect potential defects using tools like PMD, CheckStyle, Findbugs, etc.

Files, classes, and methods with poor complexity distribution can be difficult to understand and modify. Without automated unit tests, any change to a component in the program may require comprehensive regression testing.

Sonar can highlight areas in the source code with severe duplication.

Insufficient or excessive comments can decrease code readability, especially when personnel changes occur. Excessive comments can also divert developers’ attention from the code itself, contradicting the original intention. Sonar can conveniently calculate and display unit test coverage.

Through Sonar, loops can be identified, package and class dependencies can be displayed, custom architectural rules can be checked, third-party jar packages can be managed, the application of individual task rules can be evaluated using LCOM4, and coupling can be detected.

Sonar Execution Process #

  1. Developers submit their code to a code management platform such as SVN or GIT.
  2. The continuous integration tool automatically triggers a build and calls the SonarScanner to scan and analyze the project code.
  3. The analysis report is sent to the SonarQube Server for processing.
  4. The SonarQube Server processes and saves the analysis report to the SonarQube Database, which is then displayed through the user interface (UI).

Sonar Environment Requirements #

The SonarQube service instance requires at least 2GB of RAM to run effectively, with 1GB of available RAM for the operating system. The amount of required disk space depends on the amount of code being analyzed by SonarQube. SonarQube must be installed on a hard drive with good read/write performance. The data folder of Sonar contains the Elasticsearch index, which will undergo a large amount of I/O operations when the server is started and running. Therefore, good read/write disk performance will have a significant impact on the overall performance of the SonarQube service.

Installation #

There are multiple ways to install Sonar. The easiest way is to use Docker. Therefore, this installation will use the Docker method. Let’s go through the detailed process below.

To install Sonar, you need to install a database. In this case, we will use mysql5.7.

Download the mysql:5.7 image.

docker pull mysql:5.7

Start Container # 

$ docker run --name mysql5.7 -v /data/mysql5.7-data:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 -d mysql:5.7

Explanation:

  • /data/mysql5.7-data is the mount directory on the host machine.

Enter the container and create the database and user.

$ docker exec -it mysql5.7 bash

root@60253d27a5f5:/# mysql -uroot -p123456
......

mysql> 
# Create database db_sonar
mysql> create database db_sonar character set utf8 collate utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

# Create database user, both username and password are sonar
mysql> grant all privileges on db_sonar.* to 'sonar'@'%'identified by 'sonar' with grant option;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

......

The configuration for sonar to use the MySQL database is now complete.

Start the sonar container #

Download the sonarqube:6.7.5 image.

docker pull sonarqube:6.7.5

Start the sonarqube container.

$ docker run -d --name sonar --link mysql5.7 -p 9000:9000 -p 9092:9092  -v /data/sonar/conf:/opt/sonarqube/conf  -v /data/sonar/data:/opt/sonarqube/data -v /data/sonar/logs:/opt/sonarqube/logs -v /data/sonar/extensions:/opt/sonarqube/extensions -e "SONARQUBE_JDBC_USERNAME=sonar"  -e "SONARQUBE_JDBC_PASSWORD=sonar" -e "SONARQUBE_JDBC_URL=jdbc:mysql://mysql5.7:3306/db_sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance&useSSL=false" sonarqube:6.7.5

Note:

  • --link means that the sonar container connects to the mysql5.7 container by its name. When configuring the database connection, you can directly use the container name as the database address.
  • -v mounts related configurations to the /data/sonar/ directory.
  • -e specifies the parameters for connecting to the database when starting sonar.

Check after all containers start successfully # 

$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS             PORTS                                            NAMES
120cb90c2248        sonarqube:6.7.5                    "./bin/run.sh"           5 weeks ago         Up 2 seconds       0.0.0.0:9000->9000/tcp, 0.0.0.0:9092->9092/tcp   sonar
60253d27a5f5        mysql:5.7                          "docker-entrypoint..."   5 weeks ago         Up 8 minutes       0.0.0.0:3306->3306/tcp, 33060/tcp                mysql5.7

Visit the web interface at http://ip:9000.

At this point, the installation of Sonar is complete.

Configuration #

Open the web interface and click “login in”.

The default username and password are both “admin”. After the first login, there will be a tutorial to create a token.

Enter any token, for example, I entered “admin”. Set the language to Java, and the build technology to Maven, as shown below.

Click “finish” to complete.

After clicking “finish”, you will be redirected to the homepage, where you will see an English version. If you are not used to it, you can also switch to Chinese.

Localization #

Click on the top-level menu “Administration” - “marketplace”, and search for “chinese” in the arrow below. The following plugin appears, click on “install”:

Wait for a moment, if the installation fails, an error like the following will be reported:

The possible reason is that the plugin version is not compatible with the Sonar version. You can download the appropriate localization plugin from the official SonarQube Community GitHub. For example, if your version is 6.7.5, you can find that the sonar-l10n-zh-plugin-1.19 version is supported on GitHub. After downloading it and placing it in the /data/sonar/extensions/plugins directory of your host machine, restart the Sonar container.

$ pwd
/data/sonar/extensions/plugins

$ wget https://github.com/SonarQubeCommunity/sonar-l10n-zh/releases/download/sonar-l10n-zh-plugin-1.19/sonar-l10n-zh-plugin-1.19.jar

$ docker restart sonar

Note:

  • When deploying Sonar, a host directory is mounted, so you just need to place the plugin in the host directory.

After the startup is complete, reopen the page and go to the application market to find the newly installed plugin.

At this point, the Sonar configuration is basically completed. Next, let’s find a piece of code to test if Sonar works well.

Test #

Here, taking scanning a Maven project as an example, go to the root directory of the project and execute the scanning command:

$ mvn sonar:sonar -Dsonar.host.url=http://sonar_ip:9000 -Dsonar.login=7e73499a981bf49e2f724d11479623a00cf97f1a

Note:

  • The value specified in sonar.login is the token generated when you just logged in.

What to do if you forget the token, just generate a new one, as shown below, first revoke it, then generate it again:

Token

After successful execution, some of the log files are as follows:

66230 [INFO] Analysis report generated in 305ms, dir size=790 KB
66816 [INFO] Analysis reports compressed in 584ms, zip size=408 KB
67682 [INFO] Analysis report uploaded in 865ms
67686 [INFO] ANALYSIS SUCCESSFUL, you can browse http://192.168.176.160:9000/dashboard/index/com.hivescm:fw-base-nop
67687 [INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
67687 [INFO] More about the report processing at http://192.168.176.160:9000/api/ce/task?id=AXBJZsjiw7C6SYqgng-6
67816 [INFO] Task total time: 44.679 s
68175 [INFO] ------------------------------------------------------------------------
68176 [INFO] BUILD SUCCESS

The interface is as follows:

Interface

At this step, Sonar can be used basically. After completing the basic scan, code analysis is still needed, which requires other plugins. The most commonly used one is, of course, SonarQube Scanner. The installation and configuration will be performed below.

SonarQube Scanner Code Analysis

Download the required installation package from the official website, unzip it, and put it in the specified directory (it is recommended to put the scanner on the same host as Jenkins, because it needs to be configured with Jenkins to work). Then add the environment variable, for example, like mine:

$ cat /etc/profile
PATH=$PATH:/opt/sonar-scanner-4.2.0/bin
export PATH

$ source /etc/profile

Then modify the global configuration:

$ cat sonar-scanner-4.2.0/conf/sonar-scanner.properties 
#Configure here general information about the environment, such as SonarQube server connection details for example
#No information about specific project should appear here

#----- Default SonarQube server
sonar.host.url=http://192.168.176.160:9000

#----- Default source code encoding
#sonar.sourceEncoding=UTF-8

After the environment variable takes effect, check if the configuration is successful:

$ sonar-scanner -h
INFO: 
INFO: usage: sonar-scanner [options]
INFO: 
INFO: Options:
INFO:  -D,--define <arg>     Define property
INFO:  -h,--help             Display help information
INFO:  -v,--version          Display version information
INFO:  -X,--debug            Produce execution debug output

After successful configuration, test it with a project.

Testing #

There are two ways to perform the testing. One is to execute directly with command line arguments, and the other is to execute by creating a configuration file in the project directory. The second method will be used for this test.

In the project root directory, create a configuration file sonar-project.properties (some parameters are not listed):

$ cat sonar-project.properties

# must be unique in a given SonarQube instance

sonar.projectKey=fw-base-nop

# this is the name and version displayed in the SonarQube UI. Was mandatory prior to SonarQube 6.1.

sonar.projectName=fw-base-nop

sonar.projectVersion=1.0

# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows.

# This property is optional if sonar.modules is set. 

sonar.sources=.

sonar.java.binaries=./target/classes

# Encoding of the source code. Default is default system encoding

sonar.sourceEncoding=UTF-8

Note:

  • sonar.projectKey: Project name, must be unique within each project
  • sonar.projectName: Project name, displayed on the SonarQube web interface
  • sonar.projectVersion: Project version
  • sonar.sources: Project directory to be scanned
  • sonar.host.url: URL for accessing the Sonar service, not specified here as it is already configured in the file
  • sonar.login: Token corresponding to the token name, not specified here either
  • sonar.java.binaries: Project compilation directory, for example, for Java, it is the directory where the class files are located
  • sonar.sourceEncoding: Specified character encoding

For more information about the parameters, please refer to here.

After a successful execution, the partial results will be shown as follows:

......
WARN: This may lead to missing/broken features in SonarQube
INFO: 37 files had no CPD blocks
INFO: Calculating CPD for 80 files
INFO: CPD calculation finished
INFO: Analysis report generated in 702ms, dir size=541 KB
INFO: Analysis reports compressed in 447ms, zip size=259 KB
INFO: Analysis report uploaded in 480ms
INFO: ANALYSIS SUCCESSFUL, you can browse http://192.168.176.160:9000/dashboard/index/fw-base-nop
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://192.168.176.160:9000/api/ce/task?id=AXBMjjIlw7C6SYqgng_B
INFO: Task total time: 14.546 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 17.906s
INFO: Final Memory: 14M/60M

Then a new project will appear on the Sonar web interface, as shown below-

At this point, the basic configuration of Sonar has been completed. However, although the test was successful, all operations were still performed in the shell bash. Therefore, the next step is to configure the integration with Jenkins.

Configuring Jenkins Integration with Sonar Scanner #

The Jenkins plugin for integrating Sonar and Scanner has already been installed at the beginning of the article. Here, we will directly demonstrate how to configure it.

Click on the menu “manage jenkins” –> “configure system” to find the “SonarQube servers” configuration parameters. Configure the connection to SonarQube as shown below:

The red box in the image above is the token credential ID used by Jenkins to connect to SonarQube. There are two ways to create this credential. The first method is to click “Add” here, select “jenkins” in the popped-up box, and:

  • For the “Kind” parameter, select Secret text.
  • For the “Secret” parameter, fill in the token created during the SonarQube installation.
  • The “ID” and “Description” fields are used to customize the name of this credential. The ID must be unique, as Jenkins calls this credential based on the ID.
  • Other parameters can be kept as default.

As shown below, after adding these details, click “Add” to add the credential. Then, select this credential from the drop-down box in the red box in the above image.

The second method is to click “Credentials” –> “System” –> “Global credentials” –> “Add Credentials” on the Jenkins home page, as shown below:

The configuration process is the same as the previous method.

Jenkins has now been configured with SonarQube. Next, configure the integration of Jenkins with SonarQube Scanner.

On the Jenkins home page, click “manage jenkins” –> “Global Tool Configuration”. Find the value for SonarQube Scanner and add the path to Scanner on the local machine, giving it a name, as shown below:

With this, the integration configuration of Jenkins with Scanner is complete. Now, let’s create a Jenkins project to test it.

Create a Maven project and configure the Maven build settings. In the “Post Steps” section, click “Execute SonarQube Scanner” and configure as follows:

There’s no need to explain the configuration parameters as we have already covered them. However, here, some parameter values are specified using variables, which can be further understood by referring to ${jenkins_url}/pipeline-syntax/globals. The Additional arguments specify the -X parameter to print detailed log information during the scanner execution.

The execution result is as shown below:

This concludes the simple introduction to configuring Jenkins integration with Sonar in this chapter. In the upcoming chapters, we will continue to explore how to use Sonar Scanner in a pipeline.