03Devops Toolchain #
Introduction to Basic Tools Installation and Configuration #
In the previous two chapters, we briefly introduced the installation and configuration of Jenkins and plugins. Although this series of articles is based on Jenkins, implementing CI/CD in DevOps is not possible with just one tool. It requires the collaboration of many DevOps toolchains. So after configuring the Jenkins service, this chapter will introduce several DevOps toolkits that integrate with Jenkins for continuous delivery and continuous deployment. In future chapters, we will use Jenkins to build and deploy application services.
This section mainly introduces the installation and configuration of the following tools:
- Docker
- Harbor
- Gitlab
- Ansible
Docker #
Introduction #
Docker is a well-known advanced container engine built on the LXC technology. It uses containers as the basic unit for resource separation and scheduling, encapsulating the entire software runtime environment. It is designed for developers and system administrators to build, deploy, and run distributed applications. Docker is a cross-platform, portable, and user-friendly container solution.
Docker is based on Linux kernel’s cgroup, namespace, AUFS, and Union FS technologies. It encapsulates and isolates processes, making it a lightweight virtualization technology at the operating system level.
Explanation #
Namespace: is a kind of encapsulation and isolation of global system resources, which allows processes in different namespaces to have independent global system resources. Changing system resources in one namespace only affects processes in the current namespace and has no effect on processes in other namespaces. The Linux kernel provides 7 types of namespace isolation system calls (Docker uses 6 of them):
UTS: UTS namespace provides isolation of hostnames and domain names.
IPC: Inter-process communication, involving IPC resources such as semaphores, message queues, and shared memory.
PID: PID namespace isolates process PIDs. Processes in different namespaces can have the same PID, and each PID namespace has its own set of PID programs. The first process in the PID namespace is PID 1 (the parent process of all processes), similar to the init process in Linux.
mount: mount supports the isolation of file system mount points for isolated file systems.
Network: Network namespace mainly provides isolation of network resources (not real network isolation, just separate the network).
User: User namespace mainly isolates security-related identifiers and attributes, such as user IDs, group IDs, root directories, keys, and permissions.
cgroup: is a mechanism provided by the Linux kernel to monitor and limit a group of processes. Cgroup can limit and record the physical resources (such as CPU, memory, I/O, etc.) used by task groups. Its main functions are as follows:
Resource limitation: limit the total amount of resources used by tasks.
Priority allocation: control the priority of task execution.
Resource statistics: statistics on the usage of system resources.
Task control: operations such as task suspension and recovery.
AUFS and Union FS:
UnionFS is a file system service designed for Linux operating systems to union multiple file systems into the same mount point.
AUFS, which stands for Advanced UnionFS, is an upgraded version of UnionFS that provides better performance and efficiency.
As a union file system, AUFS can unionize (Union) layers from different folders into the same folder. These folders are called branches in AUFS, and the entire union process is called union mount.
This is a brief introduction to Docker. Next, let’s take a look at Docker installation.
Installation #
Install Docker on Centos7.x
Before installation, uninstall the old version of Docker service (if you have installed it before):
$ sudo yum remove docker docker-common docker-selinux docker-engine docker-client*docker-latest* docker-logrotate
# Remove remaining files
$ rm -rf /var/lib/docker
Install repository packages:
$ yum install -y yum-utils device-mapper-persistent-data lvm2
Set up stable repository:
$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Note: The official source installation can be slow. You can use Aliyun as an alternative:
https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
Install Docker CE:
$ yum-config-manager --enable docker-ce-edge && yum install docker-ce -y
Start Docker:
$ systemctl start docker
Automatic Installation Using Script
Docker provides an installation script to simplify the installation process. You can use this script on CentOS systems:
$ curl -fsSL get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh --mirror Aliyun
Note:
--mirrorspecifies the yum source.- After running this command, the script will automatically prepare everything and install Docker on the system.
By default, the docker command communicates with the Docker engine using the Unix socket. And only users in the root group or the docker group can access the Docker engine’s Unix socket. For security reasons, the root user is usually not directly used on Linux systems. Therefore, it is better to add the user who needs to use Docker to the docker group (if you don’t want to use the root user).
Create docker group:
$ sudo group add docker
Add the current user to the docker group:
$ sudo usermod -aG docker $USER
For installing Docker on other systems, you can refer to the official website, such as
Ubuntu installation:
https://docs.docker.com/install/linux/docker-ce/ubuntu/
Mac installation:
https://docs.docker.com/docker-for-mac/install/
The installation is relatively straightforward.
Test #
After installing Docker, you can proceed to the basic test.
Here is an example of starting a container using the official nginx image:
[root@Docker ~]# docker run -itd -p80:80 nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
bc51dd8edc1b: Pull complete
.....
5c472411317e6f1f13ed522d3963331054c84f747cd88d66f52d67be66709973
After starting, check and test the access:
[root@Docker ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
72dd0bd9a12f nginx "nginx -g 'daemon ..." 5 seconds ago Up 2 seconds 0.0.0.0:80->80/tcp gallant_nash
[root@Docker ~]# curl localhost:80
......
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
......
</body>
</html>
When using the docker run command to start a container, if the image does not exist locally, it will be pulled from Docker Hub by default. The speed of pulling the image may be slow due to the slow access to Docker Hub from the domestic network. In this case, a mirror accelerator needs to be configured for the Docker service to improve the speed of image pulling.
For the CentOS system, edit the /usr/lib/systemd/system/docker.service file, find the ExecStart= line, and add --registry-mirror="accelerator address" at the end of this line, such as:
ExecStart=/usr/bin/dockerd --registry-mirror=https://jxus37ad.mirror.aliyuncs.com
Or use the official Docker accelerator:
https://registry.docker-cn.com
Or use the official method:
# Official Docker image accelerator, if the file does not exist, create a new file and make sure the file conforms to the JSON format
cat /etc/docker/daemon.json
{
"registry-mirrors": [
"https://registry.docker-cn.com"
]
}
After configuring, you need to restart the Docker service:
systemctl daemon-reload
systemctl restart docker
Execute docker info in the command line. If you see the added accelerator content in the result, it means the configuration is successful. For example:
Registry Mirrors: https://registry.docker-cn.com/
That’s all for the installation configuration of Docker. So, what can the integration of Jenkins and Docker do? According to the plugins provided by Jenkins official, there are mainly two functionalities: using Docker as the deployment environment for application services, and using Docker as the dynamic slave node for Jenkins. In actual work, the commonly used functionality is using Docker as the dynamic slave node for Jenkins to improve the performance and efficiency of Jenkins services. The detailed introduction on how to integrate and use it will be provided in future chapters.
Harbor #
Before introducing Harbor, it is necessary to understand the concept of Docker image repository. The image repository is a centralized place to store images.
A concept that can be easily confused is the registry server. In fact, the registry server is the specific server that manages the repository. Each server can have multiple repositories, and each repository can have multiple images. From this perspective, a repository can be considered as a specific project or directory. For example, for the repository address dl.dockerpool.com/ubuntu, dl.dockerpool.com is the address of the registry server and ubuntu is the repository name.
Most of the time, it is not necessary to strictly distinguish between these two concepts.
Harbor is an enterprise-level registry server for storing and distributing Docker images. Harbor supports image replication across multiple registry nodes, and all images are stored in private registries. As an enterprise-level private registry server, Harbor provides better performance and features such as user management, access control, and activity auditing. It improves the efficiency for users to use the registry to build and run environments and transfer images.
Harbor Components #
Harbor is mainly composed of six components in its architecture:
Proxy: The registry, UI, token, and other services of Harbor. It receives requests from browsers and Docker clients through a reverse proxy and forwards the requests to different backend services.
Registry: Responsible for storing Docker images and handling docker push/pull commands. As we need to control user access, which means different users have different read and write permissions to Docker images, the registry points to a token service. It enforces that every docker pull/push request from the user must carry a valid token, which the registry verifies by decrypting it with a public key.
Core services: These are the core functionalities of Harbor and mainly provide the following services:
UI: Provides a graphical interface to help users manage images on the registry and authorize users.
Webhook: Configured on the registry to receive notifications of image status changes so that the information can be passed on to the UI module.
Token service: Responsible for issuing tokens to docker push/pull commands based on user permissions. If a request from the Docker client to the Registry service does not contain a token, it will be redirected to this service, which will issue a token and then initiate the request to the Registry service again.
Database: Provides database services to the core services, responsible for storing user permissions, audit logs, Docker image grouping information, and other data.
Job Services: Provides image replication functionality, allowing local images to be synchronized to other Harbor instances.
Log collector: Responsible for collecting logs from other components to facilitate monitoring the running of Harbor and for future analysis.
Each component of Harbor is built as a Docker container. The official deployment of Harbor utilizes Docker Compose. The Docker Compose template for deploying Harbor is located in harbor/docker-compose.yml. After the installation of Harbor, you can use the docker ps command or docker-compose ps command to see that Harbor consists of 7 containers (or 9 containers in newer versions), as shown below:
# docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------------------------------------------
harbor-adminserver /harbor/harbor_adminserver Up
harbor-db docker-entrypoint.sh mysqld Up 3306/tcp
harbor-jobservice /harbor/harbor_jobservice Up
harbor-log /bin/sh -c crond && rm -f ... Up 127.0.0.1:1514->514/tcp
harbor-ui /harbor/harbor_ui Up
nginx nginx -g daemon off; Up 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->80/tcp
registry /entrypoint.sh serve /etc/ ... Up 5000/tcp
Explanation
nginx: Nginx is responsible for traffic forwarding and security authentication. All external traffic flows through Nginx, so ports 443 for HTTPS are open. It distributes the traffic to the UI and the Docker registry that stores the Docker images.
harbor-jobservice: Harbor-jobservice is the job management module of Harbor. In Harbor, jobs are primarily used for synchronizing between image repositories.
harbor-ui: Harbor-ui is the web management interface, which mainly consists of the frontend pages and backend CURD (Create, Update, Read, Delete) APIs.
registry: The registry is the native Docker registry, responsible for storing images.
harbor-adminserver: Harbor-adminserver is the system management interface of Harbor, through which system configurations can be modified and system information can be obtained.
harbor-db: Harbor-db is the database of Harbor, which stores job information, project, and user permission management. Since the authentication in this Harbor is also based on the data, it is often integrated with the LDAP of the enterprise in the production environment.
harbor-log: Harbor-log is the logging service of Harbor, which manages the logs generated by Harbor. From the inspect command, it can be observed that the containers uniformly output the logs to syslog.
These containers are connected to each other through Docker links, allowing them to access each other by container names. For end-users, only the service port of the proxy (Nginx) needs to be exposed. Other component usages only require a simple understanding.
Installation and Configuration #
Download the offline version of Harbor:
https://github.com/goharbor/harbor/releases
Configure Harbor:
$ wget https://storage.googleapis.com/harbor-releases/release-1.7.0/harbor-offline-installer-v1.7.5.tgz
$ tar xvf harbor-offline-installer-v1.7.5.tgz
$ cd harbor
Modify the contents of harbor.cfg as follows:
# IP address or hostname for accessing the management UI and registration service, do not use localhost or 127.0.0.1 as Harbor needs to be accessed by external clients
hostname = 192.168.176.155
# Modify the administrator password
harbor_admin_password = Harbor12345
Only the harbor address and login password need to be modified. Prior to that, Docker Compose needs to be installed:
$ sudo curl -L https://github.com/docker/compose/releases/download/1.17.1/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
Execute ./install.sh to automatically download the images and start the process.
After starting, access Harbor at http://192.168.176.155/harbor/sign-in with the user/password admin/Harbor2345.
At this point, Harbor is installed and configured.
Commands to stop or start Harbor:
# In the harbor directory
# Start in the foreground
docker-compose up
# Start in the background
docker-compose up -d
# Stop
docker-compose down
By default, Docker does not allow image pushing over non-HTTPS connections. However, this restriction can be lifted by modifying Docker’s configuration options.
$ cat /usr/lib/systemd/system/docker.service |grep dockerd
ExecStart=/usr/bin/dockerd -H unix:// --insecure-registry 192.168.176.155
$ docker login 192.168.176.155
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
From the login information above, it can be seen that the credentials for Docker’s private repository are stored in the /root/.docker/config.json file.
Test pushing an image:
$ docker tag nginx 192.168.176.155/library/nginx:test
$ docker push 192.168.176.155/library/nginx:test
The push refers to repository [192.168.176.155/library/nginx]
332fa54c5886: Pushed
6ba094226eea: Pushed
6270adb5794c: Pushed
test: digest: sha256:e770165fef9e36b990882a4083d8ccf5e29e469a8609bb6b2e3b47d9510e2c8d size: 948
Login to harbor to view:
At this point, the basic installation and configuration of Harbor is complete.
GitLab #
Introduction #
GitLab is an open-source project for managing code repositories. It uses Git as the code management tool and is built on top of it to provide web services.
GitLab is also a popular and free code hosting service used by many companies. In addition to being a source code repository service, GitLab also has another important feature: GitLab-CI, a continuous integration system based on GitLab. Starting from GitLab 8.0, the CI feature (GitLab CI) is integrated by default. It reads the CI tasks from the .gitlab-ci.yaml configuration file created within the project and carries out continuous delivery and deployment operations through an agent called GitLab Runner. Whenever there is a code update in the GitLab repository, the predefined script (.gitlab-ci.yaml) will be triggered to perform the preset operations, such as code compilation, testing, and deployment.
Installation #
Installing GitLab is relatively simple. The official GitLab website provides detailed installation steps for different systems, which can be found here.
For convenience in testing, I will be using Docker to install GitLab. Use the following command to start the container:
docker run -d -p 4443:443 -p 80:80 -p 2222:22 --name gitlab --restart always -v /srv/gitlab/config:/etc/gitlab -v /srv/gitlab/logs:/var/log/gitlab -v /src/gitlab/data:/var/opt/gitlab gitlab/gitlab-ee
Note:
The GitLab image by default uses port 80. When deploying GitLab using Docker and setting start parameters, it is recommended that you also use port 80 for the host machine (i.e. -p80:80 parameter). Do not change this randomly, as it may cause unexpected issues. For example:
When installing GitLab using Docker, I encountered a problem (which didn’t exist when using a VM). This was because the mapped local port was different from the internal port used by GitLab. As a result, there were network connection issues when using the GitLab Runner container to fetch application code, as shown below:
Note:
By default, when the GitLab Runner fetches code, it uses the internal IP or hostname of the GitLab service container to do so. This will definitely cause network connection issues. Therefore, when port mapping, it is necessary to ensure that the host machine and the container use the same port. Since the default internal port for GitLab is 80, it is recommended to use port 80 for the host machine as well.
What if you want to use a different port? For example, if you use -p8099:80, you will encounter a process of troubleshooting. Although modifying the GitLab configuration file, restarting the container and the service can solve the aforementioned problem, there are many pitfalls in this process. For example, although you can fetch code after modifying the address and port, it may affect the operation of GitLab-CI pipelines, or the container may revert to the default configuration after restarting. Therefore, for safety reasons, it is best to use port 80 for the host machine when doing port mapping.
After creating a project in GitLab, when retrieving the project URL from the project settings, you will notice that the address is http://$container_id/$group/$project_name.git. Although you know the actual address of GitLab, each time you create a project, the host in the project address is the ID of the container, and you have to manually replace it when copying and using it. This is unacceptable for rigorous operations and maintenance. Therefore, it is necessary to modify the URL address of the projects.
After the container starts, modify the /srv/gitlab/config/gitlab.rb file on the host machine or the /etc/gitlab/gitlab.rb file in the container. Find the external_url keyword, uncomment it, and change it to the IP address of the host machine (which will be used to change the address for fetching code in newly created projects).
external_url 'http://192.168.176.154'
Then restart GitLab:
Two methods (either one):
-
Enter the container and run
gitlab-ctl restart. -
Or restart the container using
docker restart gitlab.
Test #
Create a project, for example, the project we want to test.
After clicking create project, you will be redirected to the following interface.
This page indicates that the project has been created successfully and provides instructions for the first code submission. These steps apply to three scenarios. The first scenario is when the file to be submitted to the code repository does not exist, so we need to create a directory ourselves and create a file under that directory to upload to the repository. Refer to steps one and two. If our project files exist and are not a Git repository, then refer to steps one and three. If we want to upload an existing project that is already a Git repository, then refer to steps one and four.
I am using the second scenario here, directly uploading an existing project to the repository. As shown below:
[root@es-165 helloworldweb]# git config --global user.name "Administrator"
[root@es-165 helloworldweb]# git config --global user.email "[[email protected]](/cdn-cgi/l/email-protection)"
[root@es-165 helloworldweb]# git init
git remote add origin http://192.168.176.154/root/test-helloworld.git
Initialized empty Git repository in /opt/helloworldweb/.git/
[root@es-165 helloworldweb]# git remote add origin http://192.168.176.154/root/test-helloworld.git
[root@es-165 helloworldweb]# git add .
[root@es-165 helloworldweb]# git commit -m "Initial commit"
[master (root-commit) 8cd1730] Initial commit
6 files changed, 76 insertions(+)
create mode 100644 README.md
create mode 100644 pom.xml
create mode 100644 src/main/webapp/.index.jsp.un~
create mode 100644 src/main/webapp/WEB-INF/web.xml
create mode 100644 src/main/webapp/index.jsp
create mode 100644 src/main/webapp/index.jsp~
[root@es-165 helloworldweb]# git push -u origin master
Username for 'http://192.168.176.154': root
Password for 'http://[[email protected]](/cdn-cgi/l/email-protection)':
Counting objects: 12, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (12/12), 1.88 KiB | 0 bytes/s, done.
Total 12 (delta 0), reused 0 (delta 0)
To http://192.168.176.154/root/test-helloworld.git
* [new branch] master -> master
Branch master set up to track remote branch master from origin.
Then, on other machines, you can test directly by using the git clone command.
Ansible #
After introducing Docker and GitLab above, now let’s briefly introduce and install a popular automation and operation tool called Ansible.
Introduction #
Ansible is an open-source, distributed, clientless, and lightweight automation configuration management tool developed using Python paramiko (a Python implementation of SSH protocol library). Ansible works based on modularization and does not have the capability of batch deployment itself. It only provides a framework, and the modules that Ansible runs truly have the ability of batch deployment. Ansible implements functions such as batch system configuration, batch program deployment, and batch command execution. The configuration syntax uses YAML and Jinja2 template language.
Installation #
Ansible is a agentless automation tool that manages computers by default using the SSH protocol. After installation, Ansible does not add a database and does not start or continue to run a daemon. It can be installed on one computer and used to manage remote computers from that central computer.
Prerequisites:
Currently, Ansible control nodes can run on any computer with Python 2 (version 2.7) or Python 3 (version 3.5 and above) installed. This includes Red Hat, Debian, CentOS, macOS, any BSD, etc. Control nodes are not supported on Windows.
Before installation, dependencies need to be resolved:
yum -y install python-jinja2 PyYAML python-paramiko python-babel python-crypto
Installation via tar ball:
Download Ansible from https://github.com/ansible/ansible/releases and extract the installation files.
python setup.py build
python setup.py install
Installation via yum:
$ yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ sudo yum install ansible
For more platforms to install, refer to the official website
After installation, execute the following command to check if the installation is successful:
$ ansible --version
ansible 2.7.6
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
If the above content appears, it means the Ansible installation was successful.
Modify the host_key_checking parameter value (remove the comment, change true to false) in the ansible configuration file used by default obtained from the above command (config file parameter):
host_key_checking = False
This is used to solve the issue of validating the _host_ server when ansible connects for the first time.
Testing #
First, you can use the default localhost group for a ping operation, for example
[root@ansible ~]# ansible localhost -m ping
localhost | SUCCESS => {
"changed": false,
"ping": "pong"
}
Ansible achieves configuration management, application deployment, task execution, and other functionalities through ssh. Therefore, to use Ansible, you need to first configure the Ansible server to connect to the managed nodes via passwordless authentication (username and password can also be used, but this section does not demonstrate it for now). Here is an example:
After generating the key pair with ssh-keygen -t rsa -P '' on the Ansible server, execute the following command:
ssh-copy-id -i ~/.ssh/id_rsa.pub <managed host IP>
After completion, create the /etc/ansible/hosts file and add the IP address of the host that has just been set up with passwordless authentication. For example:
$ cat /etc/ansible/hosts
[ansible_ag1] # Customize this name
192.168.177.43 # IP address of the host to be remotely managed
Then execute the following command:
$ ansible ansible_ag1 -m shell -a "ifconfig"
192.168.177.43 | CHANGED | rc=0 >>
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
inet6 fe80::42:91ff:fea9:955c prefixlen 64 scopeid 0x20<link>
......
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.177.43 netmask 255.255.254.0 broadcast 192.168.177.255
......
The above Ansible command uses the shell module specified by the -m parameter to execute the command “ifconfig” on the remote host through the -a parameter.
After testing Ansible without any issues, we can proceed to configure the Ansible environment variables in Jenkins. Click on “Manage Jenkins” -> “Global Tool Configuration”, find the “Ansible” options, click “Add Ansible” and configure the environment variables for the Ansible command, as shown below:
Once configured, click save.
That’s all for the installation and configuration of basic tools. The content is relatively simple, and the methods for using these tools will be introduced in detail in future chapters.