How Is the System of Our Course Built

How is the system of our course built #

Hello, I’m Gao Lou.

In our course, in order to help you better understand my performance engineering concepts, I have specifically built a complete system. All of our content is based on this system.

Since the course was updated, many students have asked me for the tutorial on building this system, wanting to try it themselves. Therefore, I have compiled a version of the tutorial in order to help you.

Due to the relative complexity of the entire system, there are many details that need to be considered and deployed. Therefore, the content of this lesson will be relatively long. The following diagram is the table of contents for this lesson. You can have an overview and then follow this table of contents to learn the specific steps of building the system in order to avoid getting lost.

1. Physical Servers #

1. Server Configuration #

In this system, we mainly use four servers with the following hardware configurations:

As we can see, the total CPU resources used in the application by the current servers amount to 64C, along with 128GB of memory resources. We do not include the NFS (Network File System) server as part of the application.

Since each machine has relatively abundant hardware resources, we will convert them into virtual machines for easier application management in the subsequent work.

In terms of cost, the total expense of all physical servers is approximately 80,000 yuan, which includes various miscellaneous expenses such as switches, cabinets, and network cables.

2. Server Setup #

Currently, the mainstream Linux systems based on x86 architecture in the industry are mainly CentOS and Ubuntu. For our project, I have chosen the CentOS series to build the Linux system, primarily due to its stability. CentOS is a recompiled version of the commercial release of Red Hat and has a relatively comprehensive testing and release process in terms of stability, system optimization, and compatibility.

In versions of CentOS after CentOS 7, the CentOS kernel has been changed to Linux 3.x. Therefore, the analysis in this course is based on the Linux 3.x kernel version.

During the setup process, we installed CentOS 7.8 operating system on each server. If you’re a beginner, I recommend using a system with a GUI desktop for easier management and operation of the virtual machines. You can refer to this link for specific steps on how to install the operating system on HP servers: Installing CentOS 7 on HP Servers.

2. Virtualization #

1. Virtual Machine Planning #

Next, let’s look at the virtual machine planning. We have deployed at least five virtual machines and classified them into two types of host node:

  • General node:

General nodes are used for non-test systems, such as stress machines and management platforms. We can choose to deploy them using Docker, binary, or other methods.

  • Kubernetes node:

Kubernetes nodes are used to deploy application services for projects, including mall-admin, mall-portal, mall-gateway, mall-member, mall-cart, and other application services, as well as basic components such as SkyWalking and Nacos. These are deployed using Kubernetes.

For the specific node planning, you can refer to this table:

Here, we have planned three Kubernetes control nodes, which are prepared for the subsequent high availability solution. If you plan to build a single master cluster, only one Kubernetes control node is needed. As for Kubernetes compute nodes, based on the previous node planning, we have configured 9 worker nodes here, and other nodes can be flexibly expanded according to your needs.

2. Virtual Machine Installation #

When it comes to installing virtual machines, we ultimately choose a solution mainly based on KVM. This is mainly because KVM is a relatively mature open-source virtualization platform that was included in the Linux kernel in 2006. Starting from RedHat 6, RedHat began to support KVM instead of the previously heavily promoted Xen virtualization solution, and Intel also began to fully support KVM. Compared to Xen, KVM is smaller, lighter, and easier to manage.

At the beginning of the project, we also tried using OpenStack as the underlying platform, but deploying OpenStack is not only complicated, but also prone to various pitfalls, requiring a significant investment of time. At that time, we spent a lot of time analyzing the problems with OpenStack itself, which was unnecessary for our system.

Therefore, we ultimately chose KVM for virtualization, as its technology is relatively mature and its operations are relatively simple.

You may wonder why we didn’t use VMware. We know that VMware is considered the best solution in terms of IO and stability in the field of virtualization platforms. It can also meet our needs. However, it is a commercial software with expensive licensing fees, which is why we had to give it up for this project. Of course, if your project has sufficient budget, VMware is a good choice.

Before installation, you can have a rough understanding of KVM’s performance, live migration, stability, application portability, and setup considerations as an extension of knowledge. For performance analysis, we need to pay attention to the optimization focus of KVM: A summary of two or three things to note about KVM virtualization

As for the installation and usage of KVM, you can refer to the content in this link: Linux KVM Installation User Manual.

3. Kubernetes Cluster #

1. Computing Resources #

Regarding computing resources in a cluster, you can refer to the following table:

When planning for computing resources, we usually need to consider different application scenarios:

  • Traditional virtualization technologies have higher I/O overhead. For I/O-intensive applications, physical machines have better performance than traditional virtual machines (such as virtual machines created by VMware’s traditional virtualization).
  • Deploying applications on physical machines incurs fewer additional resource costs (such as virtualization management, virtual machine operating systems, etc.), and allows for higher deployment density, reducing infrastructure costs.
  • On physical machines, you can choose network, storage, and other devices and software applications more flexibly.

If considering the actual production environment, the general recommendations are:

  • For applications that are highly sensitive to performance, such as high-performance computing, physical machines are a better choice.
  • Cloud instances support hot migration, which can effectively reduce operational costs.
  • In practice, we divide the Kubernetes cluster into a static resource pool and an elastic resource pool. Typically, physical machines or cloud instances can be selected for the fixed resource pool as needed. The elastic resource pool can use cloud instances of appropriate specifications based on application workload to optimize costs, avoid resource waste, and enhance elastic supply guarantees.

Since our system is just an example project for the course and to make the most of server resources and save server costs, we choose to prepare virtual machines on our own, which allows for the full utilization of hardware resources.

2. Cluster Setup #

Regarding cluster setup, our node planning is as follows:

For the specific steps of cluster setup, you can follow the deployment instructions in the following two documents:

The load balancing components to be installed are as follows:

If you don’t have experience with Kubernetes, I recommend reading the following introductory articles:

3. Plugin Installation #

We need to install three types of plugins: network plugins, storage plugins, and components.

For network plugins, we use Calico, which is currently the mainstream network plugin. If you have other requirements for the selection of network plugins, you can refer to the article below. I won’t go into details here.

The specific steps for installing the Calico plugin are already explained in the previous document on deploying a single master cluster, so you can refer to it. For the storage plugin, we chose NFS network storage. Because NFS is relatively simple and easy to get started with, we just need to deploy an NFS service, use a Kubernetes auto-configured volume program, and then dynamically configure PVC through StorageClass. Moreover, in terms of performance, NFS can also meet the needs of our system.

However, NFS is not a high availability solution. If you are using it in a production environment, you can consider using Ceph as a storage option. Ceph is a unified distributed storage system and a high availability storage solution that can provide good performance, reliability, and scalability. However, deploying Ceph is more complex and maintaining it is also more complex than NFS.

I have provided the detailed installation steps for NFS and Ceph here for your reference if needed.

Also, don’t forget that two components are required in the NFS configuration:

4. Kubernetes Management Platform #

Installation components:

Kuboard uses a visual UI to manage applications and components, reducing the learning curve of Kubernetes clusters. Let’s see how to deploy the Kuboard component.

Step 1, execute the resource files on the k8s cluster:

kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
kubectl apply -f https://addons.kuboard.cn/metrics-server/0.3.7/metrics-server.yaml

Step 2, after Kuboard is installed, let’s check its running status:

kubectl get pods -l k8s.kuboard.cn/name=kuboard -n kube-system

Output:

NAME                       READY   STATUS        RESTARTS   AGE
kuboard-54c9c4f6cb-6lf88   1/1     Running       0          45s

This result indicates that Kuboard has been successfully deployed.

Next, we need to obtain the administrator token. This step is for logging in to access Kuboard and checking if the components are running successfully.

# Execute this command on the first Master node
echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d)

By checking the deployment, we know that the Kuboard Service exposes the service using NodePort, with NodePort set to 32567. Therefore, we can access Kuboard in the following way:

http://<Worker node IP address>:32567/

Then, enter the administrator token during login to access the overview page of the Kuboard cluster.

Note that if you are using cloud services such as Alibaba Cloud or Tencent Cloud, you can open inbound access to port 32567 for worker nodes in the corresponding security group settings. You can also modify the Kuboard.yaml file to use a custom NodePort port number.

IV. Dependent Components #

1. Deployment List #

2. Installation and Deployment #

For the installation and deployment of the above dependent components, I have compiled corresponding tutorials and placed them here. If you are interested, you can give it a try.

For the binary installation of MySQL, there are numerous tutorials available online, so I won’t introduce it here. If you want to know how to deploy MySQL in Kubernetes, you can refer to the detailed steps in this link: How to Build a Complex MySQL Database in a Kubernetes Cluster.

For the deployment of Elasticsearch clusters, you can refer to:

For the deployment of JMeter, you can refer to:

For the deployment of the image repository Harbor, you can refer to:

For the deployment of Nacos, you can refer to:

For the deployment of Redis, RabbitMQ, and MongoDB in standalone mode, you can refer to:

For the deployment of Logstash, you can refer to:

5. Monitoring Components #

1. Global Monitoring #

Do you remember the architecture of our system?

Based on this system architecture, we need to monitor the following levels:

  • Level 1: Physical Hosts
  • Level 2: KVM Virtual Machines
  • Level 3: Kubernetes Suite
  • Level 4: Various technical components required by applications

With the above system architecture, monitoring design has already come to the mind of the writer. For the system used in this course, the global monitoring is as follows:

From the above diagram, we can use Prometheus/Grafana/Spring Boot Admin/SkyWalking/Weave Scope/ELK/EFK to achieve first-level monitoring with a global perspective. For first-level counters not covered by these tools, we can only supplement them by executing commands during the execution scenario.

2. Deployment Manifest #

3. Installation and Deployment #

For the deployment of these monitoring tools, I have also included the corresponding installation tutorials here for your reference and learning.

Deployment of Kubernetes cluster resource monitoring:

Deployment of log aggregation:

Deployment of dependent components:

Deployment of APM trace tracking:

6. Microservices #

1. Project Introduction #

When building the system used in this course, I adopted the microservices architecture, which is also the mainstream technology architecture currently.

If you are interested in learning more about the project, you can refer to this article: 《Performance Engineering Practice of High Building》In-depth Analysis of Microservices E-commerce Project. It mainly introduces some background knowledge of the project, system structure, main technology stack, and core components. In addition, there are screenshots of the running effects.

2. Clone the Source Code #

We clone the project source code to our local machine to deploy our system under test:

git clone https://github.com/xncssj/7d-mall-microservice.git

3. Modify Nacos Configuration #

First, we import the configuration package in the project’s config directory into Nacos, and then modify the relevant configurations according to our actual needs.

After importing the configuration into Nacos, the following message will be displayed:

Please note that the configuration files we modify are mainly application-prod.yml and bootstrap-prod.yml under each monolithic service. These two global configuration files are loaded by the service container.

4. Package and Push the Image #

We use an IDE for Java (IDEA is recommended) to open the project.

First, modify the pom.xml file in the project root directory:

<properties>
    <!--Change to your own remote access address of the Docker service-->
    <docker.host>http://172.16.106.237:2375</docker.host>
</properties>

In the Maven tab on the right side of IDEA, we can find the package button under the root project, select it and execute:

Then, on the remote Docker host where the compilation is performed, we modify the image tag names of all services and then push the images to the Docker repository.

5. Import Database #

In this step, we need to import the SQL scripts in the document/sql directory of the project into the MySQL database.

6. Initialize Dependent Components #

6.1. RabbitMQ #

First, enter the RabbitMQ container and enable management features:

# When logging into the container, be aware of the supported shell in the container.
kubectl exec -it <pod-name> -n <ns-name> bash
kubectl exec -it <pod-name> -n <ns-name> sh

root@cloud-rabbitmq-5b49d784c-gbr8m:/# rabbitmq-plugins enable rabbitmq_management
Enabling plugins on node rabbit@cloud-rabbitmq-5b49d784c-gbr8m:
rabbitmq_management
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@cloud-rabbitmq-5b49d784c-gbr8m...
Plugin configuration unchanged.

Because the RabbitMQ Service uses NodePort to expose the console address, for example, NodePort is 15672. So, the next step is to access the address http://:15672/ and check if the installation is successful:

Next, enter the username and password guest/guest to log in.

Then, create an account and set its role as mall/mall administrator.

Then, create a new virtual host /mall.

Finally, go to the user configuration page by clicking on the mall user and configure the mall user’s permissions for this virtual host.

At this point, the initialization of RabbitMQ is complete.

6.2. Elasticsearch #

Install the Chinese word segmentation analyzer IKAnalyzer and restart:

# This command needs to be run inside the container.
elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.6.2/elasticsearch-analysis-ik-7.6.2.zip

7. Deploy the Application using YAML Resource Files #

Modify the Docker image in the YAML resource files in the document/k8s directory of the project to your own tag and upload them to the k8s cluster for execution:

kubectl apply -f k8s/

## 7. Demonstration of Performance

### 1. Server

![](../images/0105d4e36d084de9b5672064bb66fe3d.jpg)

### 2. Virtual Machines

![](../images/24259e389328405896893b80d1f47201.jpg)

### 3. Kubernetes Cluster

Kubernetes cluster:
    
    
    [root@k8s-master-1 ~]# kubectl get nodes
    NAME           STATUS   ROLES    AGE   VERSION
    k8s-master-1   Ready    master   26d   v1.19.2
    k8s-master-2   Ready    master   26d   v1.19.2
    k8s-master-3   Ready    master   26d   v1.19.2
    k8s-worker-1   Ready    <none>   26d   v1.19.2
    k8s-worker-2   Ready    <none>   26d   v1.19.2
    k8s-worker-3   Ready    <none>   26d   v1.19.2
    k8s-worker-4   Ready    <none>   26d   v1.19.2
    k8s-worker-5   Ready    <none>   26d   v1.19.2
    k8s-worker-6   Ready    <none>   26d   v1.19.2
    k8s-worker-7   Ready    <none>   26d   v1.19.2
    k8s-worker-8   Ready    <none>   26d   v1.19.2
    k8s-worker-9   Ready    <none>   26d   v1.19.2
    [root@k8s-master-1 ~]# 
    

Microservice management:

![](../images/50019a50d09844738007221bdf7da287.jpg)

### 4. Microservices

Deployment architecture diagram:

![](../images/64894adc6d314c12907aaa8e34fa288c.jpg)

API documentation:

![](../images/e78495037a224d38b9457713a9c08b71.jpg)

Call chain monitoring:
- ![](../images/0d5c1a1d39f2489bae6d2ddb12f7e6c4.jpg)
- ![](../images/8280b817178d466cb203f8e1083e869f.jpg)

![](../images/62a5a30292674cf88bc01369f6f5ace3.jpg)

Service registration:

![](../images/efa085195525434ca38f3a39acbfcf97.jpg)

Service monitoring:
- ![](../images/e992f02a8c344ea7ad8711e0213f4981.jpg)

Log aggregation:

![](../images/32dfd4958ac1442eb8f75f2d722a7bdc.jpg)

Configuration management:

![](../images/c3780c97b1a84f7199f5982335e01193.jpg)

System protection:

![](../images/20c13ece33e24f729dd6cefcb115a300.jpg)

Container registry:

![](../images/4280702aa6c447e7a1b92b598776e54b.jpg)

Stress engine:

![](../images/358b6a52aac844a682ffe23c429fc67c.jpg)

### 5. Resource Monitoring

Kubernetes cluster resource monitoring:

![](../images/19bc98d3839545d9961594532eaf2aba.jpg)

Linux resource monitoring:

![](../images/c1b86cccbb4e4143af0eef21befdee5e.jpg)

MySQL resource monitoring:

![](../images/70042b6420954484932c126b93b94e23.jpg)

RabbitMQ resource monitoring:

![](../images/506905151126495d901deebca081b508.jpg)

MongoDB database resource monitoring:

![](../images/6592a47a6c454b72b96b475fd79af2bb.jpg)

Kubernetes etcd resource monitoring:

![](../images/6392c1e852654e9399dc7ab489f02064.jpg)

Kubernetes API Server resource monitoring:

![](../images/89f1dcfe006647428fd658397f365bac.jpg)

Kubernetes service topology:

![](../images/844b6143e32b4f99be0b74e85f174f8e.jpg)
## 8. Conclusion

The content of this lesson includes the explanation of the physical environment, the specific process of building technical components, the process of building the example system, and the running effect. After going through all the steps, we have successfully constructed all the technical components and example systems involved in the entire course.

I chose this technology stack for three main reasons:

**1. Core advantages**

  * Task scheduling: provides scheduling services for tasks in cluster systems, automatically allocating services to computing nodes based on resource requirements;
  * Resource isolation: provides control and isolation capabilities for products, ensuring that development applications and control services do not interfere with each other;
  * High availability: automatically monitors service operation and automatically restarts failed services based on their operation status;
  * Network connectivity: provides unified IP address allocation and network connectivity capabilities;
  * Unified orchestration and management capabilities: combined with Gitlab and k8s, provides unified orchestration and management for output products;
  * Common product components can provide teams with unified deployment, verification, authorization, scheduling, and control capabilities, and provide infrastructure support for private cloud services.

**2. Core infrastructure platform (IaaS Cloud)**

  * Provides virtualization of core resource devices such as computing, networking, and storage;
  * Supports different operating systems, including mainstream Win and Linux systems;
  * Provides three main services: cloud servers, cloud networks, and cloud disks;
  * Provides a visual Web UI;
  * Provides planning, deployment, and operation of k8s clusters (container cloud);
  * Supports multiple computing, storage, and networking solutions.

**3. Basic service platform (PaaS Cloud)**

  * Provides data storage: supports common NFS, Ceph RBD, Local Volume, etc.;
  * Provides application services: supports self-healing, automatic scaling, scheduling and release, load balancing, etc.;
  * Provides operation and maintenance management: supports log monitoring, resource monitoring, message alerts, etc.

The technology stack used in our system is a popular mainstream technology stack in the current technology market, and such an environment has high reference value. Moreover, from the perspective of the RESAR performance analysis architecture and logic that we want to express, it also shows that the RESAR performance analysis concept is sufficient to support the current technology stack.
## Reference Materials Summary

**1. Deployment of CentOS 7**: [Installing CentOS 7 on HP Servers](https://blog.csdn.net/zuozewei/article/details/84951690)
- **2. Key optimization points for KVM**: [Notes on KVM virtualization](https://mp.weixin.qq.com/s/L-jNlwYKFWgoKX6L5ffz5A)
- **3. Installation and usage of KVM**: [Linux KVM Installation and Usage Manual](https://mp.weixin.qq.com/s/UluDn0jG-uYZqpIsIXzKAg)
- **4. Building a Kubernetes cluster**:

  - Single Master cluster: [Installing a single-master Kubernetes cluster using kubeadm (script version)](https://mp.weixin.qq.com/s/eQOMt3dz1P2JwezDaoZGzg)
  - High availability solution: [Landing a high availability Kubernetes cluster](https://mp.weixin.qq.com/s/bdq4GySQWjcIjJmn0ZD80g)


**5. Basics of using Kubernetes**:

  - [Basic concepts of a Kubernetes cluster](https://mp.weixin.qq.com/s/X2Z_a_eYq12O6yTQmRRxfw)
  - [Getting started with Kubernetes - Basic concepts and terminology of Kubernetes](https://mp.weixin.qq.com/s/dSbCK_ms4YMoEvqp0afNZg)
  - [Kubernetes command guide - Practical commands for working with Kubernetes](https://mp.weixin.qq.com/s/EPdOEwFaoc-hI27C0v3urA)
  - [Summary of commonly used operations in a Kubernetes cluster](https://mp.weixin.qq.com/s/xYjR5_WZNhny_0RdjM1tpA)


**6. Selection of Kubernetes network plugins**: [Benchmark test results of Kubernetes network plugins (CNI) exceeding 10 Gbit/s](https://mp.weixin.qq.com/s/H4ncgG0AqsmpPSg3ICBP3g)

**7. Deployment of NFS**: [Deploying NFS network storage in a Kubernetes cluster](https://mp.weixin.qq.com/s/xKPAHWuLBMYuaVzpel5rUQ)

**8. Deployment of Ceph**: [Deployment of Rook Ceph, a distributed storage plugin for Kubernetes clusters](https://mp.weixin.qq.com/s/tww0dPNeFNR_WaEWUh97ZQ)

**9. Deployment of MySQL in Kubernetes**: [Building a complex MySQL database in a Kubernetes cluster](https://mp.weixin.qq.com/s/4YSx4a2oqjrp1f6wNQgR-g)

**10. Deployment of Elasticsearch clusters**: [Deploying Elasticsearch & Kibana 7 clusters using Kubernetes Helm3](https://mp.weixin.qq.com/s/VAlbWoSf_16-Cf5nIwEShg)

**11. Deployment of JMeter**:

  - Binary: [Building real-time monitoring of performance testing with JMeter, InfluxDB, and Grafana](https://mp.weixin.qq.com/s/nUNAz3Gma9fwoFNFA9Orog)
  - Kubernetes: [Deploying JMeter clusters in Kubernetes](https://mp.weixin.qq.com/s/VQl05FvzK456WAkXDM6I9Q)


**12. Deployment of the Harbor image repository**: [Deploying the Harbor repository in a Kubernetes cluster using Helm3](https://mp.weixin.qq.com/s/_0hEWz7TUvYGASg9I3az4g)

**13. Deployment of Nacos**:

  - Docker single-node mode: [Quick start with Nacos Docker](https://nacos.io/zh-cn/docs/quick-start-docker.html)
  - Kubernetes: [Nacos with Kubernetes](https://nacos.io/zh-cn/docs/use-nacos-with-kubernetes.html)


**14. Deployment of Redis, RabbitMQ, and MongoDB**:

[Monitoring Kubernetes clusters using kube-prometheus auto-discovery](https://mp.weixin.qq.com/s/yuAU-qWeTr4svaSI497sbQ)

**15. Deployment of Logstash**: [Log collection with ELK integration](https://mp.weixin.qq.com/s/D7Xuw_8nsituhR33BpjU2w)

**16. Deployment of Kubernetes cluster resource monitoring**:

  - [Deployment of kube-prometheus monitoring for a Kubernetes cluster](https://mp.weixin.qq.com/s/AOl_z0zMuPzRlf36J6OLMg)
  - [Monitoring controller-manager & scheduler components of a Kubernetes cluster](https://mp.weixin.qq.com/s/T1Yhbw2efw-wjKu1PZyudg)
  - [Monitoring ETCD components of a Kubernetes cluster](https://mp.weixin.qq.com/s/chvukvjmCDvrYqYapRCEkA)


**17. Deployment of log aggregation**: [Installing EFK for log monitoring in a Kubernetes cluster](https://mp.weixin.qq.com/s/7shBCfpKRBBQDsutSpClhQ)

**18. Deployment of dependent components**: [Monitoring Kubernetes clusters using kube-prometheus auto-discovery](https://mp.weixin.qq.com/s/yuAU-qWeTr4svaSI497sbQ)

**19. Deployment of APM distributed tracing**: [Integrating SkyWalking with Kubernetes and Spring Cloud for distributed tracing](https://mp.weixin.qq.com/s/bDfPFjTF5W2z8XUD-8EYQQ)

**20. Introduction to microservice projects**: [Technical analysis of the microservice e-commerce project in "Performance Engineering Practice Course"](https://mp.weixin.qq.com/s/a8nDBbkuvSjreaCxuM2PuQ)

**21. Recommended resources for further learning**:

  - [Notes on SpringCloud logs in performance testing](https://mp.weixin.qq.com/s/JeKdirFrM5LGAqZrrdKoXA)
  - [Script development in "Performance Engineering Practice Course"](https://mp.weixin.qq.com/s/KHGfK7DUbSBcNOF6J8mb6Q)
  - [Recommended knowledge points for "Performance Engineering Practice Course" learning](https://mp.weixin.qq.com/s/tYooETSGhiMBBDlwihiqAA)