15 Query commodity resource insufficiency and performance behaviors #
Hello, I am Gao Lou.
In this lesson, we will focus on the “Querying Products” API. Although the symptoms this time are still low TPS and long response times, the path taken by this API is different from before. Therefore, there will be some new things in the analysis process. You will see that when resources are truly insufficient, the only way to improve performance is by adding resources to the corresponding nodes.
In my project, I have always emphasized that we should not easily conclude that resources are insufficient. Whenever there is room for optimization, we should attempt to optimize, rather than simply suggesting to the client to add resources. Moreover, the conclusion of “increasing resources” must be based on sufficient evidence. In this lesson, you will also see this point.
Without further ado, let’s dive into today’s content.
Stress Test Data #
For the query product API, the results of our first performance stress test are as follows:
As you can see, the TPS is only around 250, and the response time clearly increases with the increase in stress. It seems that a bottleneck has occurred, right? According to my logic, the next step is to look at the architecture diagram.
First, let’s look at the architecture diagram #
Let’s use an APM tool to analyze the architecture of this interface.
As you can see, from the load balancer to the Gateway service, to the Search service, and finally to the ES-Client, the APM tool can only help us up until here. This is because we are using ElasticSearch 7 as the support for the search service, and the Skywalking tool does not have a corresponding agent, so it is not configured after this point.
I want to add a few more words here. Most APM tools nowadays are based on the application layer. Some operations APM tools may collect more data, such as response time and throughput. In terms of performance analysis, current APM tools have the ability to reduce troubleshooting time, but there is still room for improvement in component-level refinement. Although AIOPS has been mentioned, I haven’t seen any company that dares to rely solely on AIOPS products.
In summary, when pinpointing the root cause of a problem, you can use whatever tools are available, or even analyze logs if no tools are available. Therefore, I suggest not to blindly rely on tools, but to “believe” in your own thinking.
Next, let’s break down the response time of this interface and identify the problem areas in this case.
Splitting Response Time #
“Splitting response time is just the beginning of performance analysis in the RESAR logic.” This is a statement that I have always emphasized. If performance engineers cannot even do this, then they need to study hard and keep improving.
According to the architecture diagram, we split the response time as follows:
- Response time on the Gateway service:
- Response time on the Search service:
- Response time on the ES Client:
After looking at each layer, we found that the response time for the “query product” interface is spent on the ES client. And along the path of this query, we haven’t done any complex actions on the gateway/search service.
Now that we know where the response time is being consumed, let’s locate it and see if we can optimize the TPS.
Global Monitoring #
According to Teacher Gao’s experience, let’s start with global monitoring, as it will help us be more focused. During the analysis process, it is common for people to become confused and lose direction after taking a few steps. With so much data available, it is easy to get sidetracked during analysis and end up focusing on unimportant branches. However, if you have global monitoring data in mind, your thinking will be clearer and you won’t waste time on irrelevant branches.
In this example, looking at the data from the k8s worker (also known as a node in Kubernetes, but I prefer to call it a worker in our environment to emphasize my personal preference), it seems that none of the workers have particularly high resource utilization.
Please note that when examining resource consumption in Kubernetes, it is important not to only look at the worker level, as it is not enough. A worker can run multiple pods. From the above image, although there is no resource consumption at the worker level, it is clear that time is being consumed by the ES client. Therefore, next, we need to look at the resource usage of each individual pod.
Hmm, there’s something in red. Look, two pods related to ES have their CPU usage in red. This could be interesting. Since they are related to ES, let’s sort all the ES pods.
From the data in the image above, one ES client is consuming 67% of the CPU, and two ES data pods are consuming 99% of the CPU. As ES is known to be CPU intensive, we need to focus on analyzing it next.
I want to clarify that the direction we have taken from examining worker resource utilization to this point is reasonable, as it is part of the global monitoring that I mentioned earlier.
Directed Analysis #
Now let’s take a look at ES and see which worker node it is on. The list of Pod information is as follows:
[root@k8s-master-1 ~]# kubectl get pods -o wide | grep elasticsearch
elasticsearch-client-0 1/1 Running 0 6h43m 10.100.230.2 k8s-worker-1 <none> <none>
elasticsearch-client-1 1/1 Running 0 6h45m 10.100.140.8 k8s-worker-2 <none> <none>
elasticsearch-data-0 1/1 Running 0 7h8m 10.100.18.197 k8s-worker-5 <none> <none>
elasticsearch-data-1 1/1 Running 0 7h8m 10.100.5.5 k8s-worker-7 <none> <none>
elasticsearch-data-2 1/1 Running 0 7h8m 10.100.251.67 k8s-worker-9 <none> <none>
elasticsearch-master-0 1/1 Running 0 7h8m 10.100.230.0 k8s-worker-1 <none> <none>
elasticsearch-master-1 1/1 Running 0 7h8m 10.100.227.131 k8s-worker-6 <none> <none>
elasticsearch-master-2 1/1 Running 0 7h8m 10.100.69.206 k8s-worker-3 <none> <none>
[root@k8s-master-1 ~]#
Now it’s clear. We can see that there are two ES clients, three ES data nodes, and three ES master nodes in the entire namespace.
Let’s draw a more detailed architecture diagram to keep this logic in mind:
Combining this with the resource utilization graph we saw in the global analysis, we can now identify at least two issues:
- Uneven ES client requests.
- High CPU usage on ES data nodes.
Now let’s analyze each issue one by one.
Uneven ES client requests #
From the above architecture diagram, we can see that the search service connects to two ES clients, but only one of them has high CPU usage. So, let’s check the link and see the ES service:
[root@k8s-master-1 ~]# kubectl get svc -o wide | grep search
elasticsearch-client NodePort 10.96.140.52 <none> 9200:30200/TCP,9300:31614/TCP 34d app=elasticsearch-client,chart=elasticsearch,heritage=Helm,release=elasticsearch-client
elasticsearch-client-headless ClusterIP None <none> 9200/TCP,9300/TCP 34d app=elasticsearch-client
elasticsearch-data ClusterIP 10.96.16.151 <none> 9200/TCP,9300/TCP 7h41m app=elasticsearch-data,chart=elasticsearch,heritage=Helm,release=elasticsearch-data
elasticsearch-data-headless ClusterIP None <none> 9200/TCP,9300/TCP 7h41m app=elasticsearch-data
elasticsearch-master ClusterIP 10.96.207.238 <none> 9200/TCP,9300/TCP 7h41m app=elasticsearch-master,chart=elasticsearch,heritage=Helm,release=elasticsearch-master
elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 7h41m app=elasticsearch-master
svc-mall-search ClusterIP 10.96.27.150 <none> 8081/TCP 44d app=svc-mall-search
[root@k8s-master-1 ~]#
You can see that there is one client service (the resolved VIP, accessing this service will not bypass K8s forwarding) and one client-headless service (the resolved POD IP, accessing this service will bypass K8s forwarding).
Next, let’s find out why there is an uneven distribution of requests.
By checking the ES configuration for the search service, we can see the following information:
elasticsearch:
rest:
uris: elasticsearch-client:9200
username: elastic
password: admin@123
We can see that we are using elasticsearch-client:9200 here. Let’s also check the configuration of the client service:
---
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: elasticsearch-client
meta.helm.sh/release-namespace: default
creationTimestamp: '2020-12-10T17:34:19Z'
labels:
app: elasticsearch-client
app.kubernetes.io/managed-by: Helm
chart: elasticsearch
heritage: Helm
release: elasticsearch-client
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata': {}
'f:spec':
'f:ports': {}
manager: Go-http-client
operation: Update
You can see that there is one client service (the resolved VIP, accessing this service will not bypass K8s forwarding) and one client-headless service (the resolved POD IP, accessing this service will bypass K8s forwarding).
Next, let’s find out why there is an uneven distribution of requests.
By checking the ES configuration for the search service, we can see the following information:
elasticsearch:
rest:
uris: elasticsearch-client:9200
username: elastic
password: admin@123
We can see that we are using elasticsearch-client:9200 here. Let’s also check the configuration of the client service:
---
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: elasticsearch-client
meta.helm.sh/release-namespace: default
creationTimestamp: '2020-12-10T17:34:19Z'
labels:
app: elasticsearch-client
app.kubernetes.io/managed-by: Helm
chart: elasticsearch
heritage: Helm
release: elasticsearch-client
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata': {}
'f:spec':
'f:ports': {}
manager: Go-http-client
operation: Update
You can see that there is a client service (resolved as VIP, accessing this service does not bypass K8s forwarding) and a client-headless service (resolved as POD IP, accessing this service will bypass K8s forwarding).
Let’s find out why the requests are not balanced. time: ‘2020-12-10T17:34:19Z’ name: elasticsearch-client namespace: default resourceVersion: ‘4803428’ selfLink: /api/v1/namespaces/default/services/elasticsearch-client uid: 457e962e-bee0-49b7-9ec4-ebfbef0fecdd spec: clusterIP: 10.96.140.52 externalTrafficPolicy: Cluster ports: - name: http nodePort: 30200 port: 9200 protocol: TCP targetPort: 9200 - name: transport nodePort: 31614 port: 9300 protocol: TCP targetPort: 9300 selector: app: elasticsearch-client chart: elasticsearch heritage: Helm release: elasticsearch-client sessionAffinity: None type: NodePort
Based on the above configuration, sessionAffinity is also configured as None, which means that this service does not maintain session using the client’s IP. In this environment configuration, the Type is NodePort, and the forwarding rule we configured in Kubernetes is iptables. Therefore, the service relies on the iptables rule for backend forwarding.
Next, let’s check the iptables forwarding rules.
Let’s start by looking at the rules for the ES client in iptables:
[root@k8s-master-1 ~]# iptables -S KUBE-SERVICES -t nat|grep elasticsearch-client|grep 9200
-A KUBE-SERVICES ! -s 10.100.0.0/16 -d 10.96.140.52/32 -p tcp -m comment --comment "default/elasticsearch-client:http cluster IP" -m tcp --dport 9200 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.140.52/32 -p tcp -m comment --comment "default/elasticsearch-client:http cluster IP" -m tcp --dport 9200 -j KUBE-SVC-XCX4XZ2WPAE7BUZ4
[root@k8s-master-1 ~]#
As we can see, the name of the service rule is KUBE-SVC-XCX4XZ2WPAE7BUZ4. Now let’s check the corresponding iptables rule for it:
[root@k8s-master-1 ~]# iptables -S KUBE-SVC-XCX4XZ2WPAE7BUZ4 -t nat
-N KUBE-SVC-XCX4XZ2WPAE7BUZ4
-A KUBE-SVC-XCX4XZ2WPAE7BUZ4 -m comment --comment "default/elasticsearch-client:http" -j KUBE-SEP-LO263M5QW4XA6E3Q
[root@k8s-master-1 ~]#
[root@k8s-master-1 ~]# iptables -S KUBE-SEP-LO263M5QW4XA6E3Q -t nat
-N KUBE-SEP-LO263M5QW4XA6E3Q
-A KUBE-SEP-LO263M5QW4XA6E3Q -s 10.100.227.130/32 -m comment --comment "default/elasticsearch-client:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-LO263M5QW4XA6E3Q -p tcp -m comment --comment "default/elasticsearch-client:http" -m tcp -j DNAT --to-destination 10.100.227.130:9200
The problem is that there doesn’t seem to be any load balancing configuration here (no probability parameter), and according to the iptables rule, the traffic is only forwarded to one ES client. From here, we can understand why when we monitor the system, we see one ES client with high CPU usage and the other one is idle.
However, these iptables rules are not manually configured. They are automatically generated when deploying Kubernetes. Since there is only one rule now, the traffic can only be forwarded to one POD.
So let’s try to recreate the ES POD and see if Kubernetes can generate load balancing iptables rules. After recreating, let’s check the iptables rules again:
[root@k8s-master-1 ~]# iptables -S KUBE-SVC-XCX4XZ2WPAE7BUZ4 -t nat
-N KUBE-SVC-XCX4XZ2WPAE7BUZ4
-A KUBE-SVC-XCX4XZ2WPAE7BUZ4 -m comment --comment "default/elasticsearch-client:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-IFM4L7YNSTSJP4YT
-A KUBE-SVC-XCX4XZ2WPAE7BUZ4 -m comment --comment "default/elasticsearch-client:http" -j KUBE-SEP-5RAP6F6FATXC4DFL
[root@k8s-master-1 ~]#
Oh, now we have two iptables rules, it seems that Kubernetes has generated load balancing iptables rules. The previous deployment process of the ES client might have caused issues.
In the iptables rules, the previous rule has a keyword “probability 0.50000000000”. We know that iptables rules are matched from top to bottom. Since the previous rule has a 50% random match, only 50% of the traffic will go through the first rule, and the next rule will also have a 50% random match, as there are only two rules. This way, the traffic is balanced.
Let’s continue with the stress test scenario for this interface, and we can see the following information:
It seems that the ES client is balanced, isn’t it?
Here is the corresponding TPS:
The TPS has increased by about 60.
The issue of unbalanced ES client requests has been resolved. Now, let’s take a look at the high CPU usage of the ES data single node.
High CPU Usage of ES Data #
- Phase 1: Adding a CPU
After the TPS improvement, let’s take a look at the global monitoring data.
It looks much better than before. Based on the previous analysis of the ES client, let’s first check the iptables rules for ES data:
-- Check which pods are ES data pods
[root@k8s-master-1 ~]# kubectl get pods -o wide | grep data
elasticsearch-data-0 1/1 Running 0 10h 10.100.18.197 k8s-worker-5 <none> <none>
elasticsearch-data-1 1/1 Running 0 10h 10.100.5.5 k8s-worker-7 <none> <none>
elasticsearch-data-2 1/1 Running 0 10h 10.100.251.67 k8s-worker-9 <none> <none>
-- Check the iptables rules for ES data
[root@k8s-master-1 ~]# iptables -S KUBE-SERVICES -t nat|grep elasticsearch-data
-A KUBE-SERVICES ! -s 10.100.0.0/16 -d 10.96.16.151/32 -p tcp -m comment --comment "default/elasticsearch-data:http cluster IP" -m tcp --dport 9200 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.16.151/32 -p tcp -m comment --comment "default/elasticsearch-data:http cluster IP" -m tcp --dport 9200 -j KUBE-SVC-4LU6GV7CN63XJXEQ
-A KUBE-SERVICES ! -s 10.100.0.0/16 -d 10.96.16.151/32 -p tcp -m comment --comment "default/elasticsearch-data:transport cluster IP" -m tcp --dport 9300 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.16.151/32 -p tcp -m comment --comment "default/elasticsearch-data:transport cluster IP" -m tcp --dport 9300 -j KUBE-SVC-W4QKPGOO4JGYQZDQ
-- Check the rules for 9200 (external communication)
[root@k8s-master-1 ~]# iptables -S KUBE-SVC-4LU6GV7CN63XJXEQ -t nat
-N KUBE-SVC-4LU6GV7CN63XJXEQ
-A KUBE-SVC-4LU6GV7CN63XJXEQ -m comment --comment "default/elasticsearch-data:http" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-ZHLKOYKJY5GV3ZVN
-A KUBE-SVC-4LU6GV7CN63XJXEQ -m comment --comment "default/elasticsearch-data:http" -m statistic --mode random --probability 1 -j KUBE-SEP-6ILKZEZS3TMCB4VJ
-A KUBE-SVC-4LU6GV7CN63XJXEQ -m comment --comment "default/elasticsearch-data:http" -j KUBE-SEP-JOYLBDPA3LNXKWUK
-- Check the forwarding target for the above three rules
[root@k8s-master-1 ~]# iptables -S KUBE-SEP-ZHLKOYKJY5GV3ZVN -t nat
-N KUBE-SEP-ZHLKOYKJY5GV3ZVN
-A KUBE-SEP-ZHLKOYKJY5GV3ZVN -s 10.100.18.197/32 -m comment --comment "default/elasticsearch-data:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZHLKOYKJY5GV3ZVN -p tcp -m comment --comment "default/elasticsearch-data:http" -m tcp -j DNAT --to-destination 10.100.18.197:9200
[root@k8s-master-1 ~]# iptables -S KUBE-SEP-6ILKZEZS3TMCB4VJ -t nat
-N KUBE-SEP-6ILKZEZS3TMCB4VJ
-A KUBE-SEP-6ILKZEZS3TMCB4VJ -s 10.100.251.67/32 -m comment --comment "default/elasticsearch-data:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-6ILKZEZS3TMCB4VJ -p tcp -m comment --comment "default/elasticsearch-data:http" -m tcp -j DNAT --to-destination 10.100.251.67:9200
[root@k8s-master-1 ~]# iptables -S KUBE-SEP-JOYLBDPA3LNXKWUK -t nat
-N KUBE-SEP-JOYLBDPA3LNXKWUK
-A KUBE-SEP-JOYLBDPA3LNXKWUK -s 10.100.5.5/32 -m comment --comment "default/elasticsearch-data:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-JOYLBDPA3LNXKWUK -p tcp -m comment --comment "default/elasticsearch-data:http" -m tcp -j DNAT --to-destination 10.100.5.5:9200
[root@k8s-master-1 ~]
Everything is perfect! The rules are reasonable. There are a total of three ES data pods, and logically, each of them occupies one third.
In the previous analysis of ES clients, we mentioned that the first pod is 0.5, so the remaining one is naturally 0.5, which is easy to understand. Now, there are three iptables rules for ES data, and we will briefly explain them.
Usually, we understand iptables as a firewall. However, fundamentally speaking, it is not a firewall, but just a list of rules. The requests in the iptables rules are matched to the netfilter framework, which is the real firewall. Netfilter is in the kernel, while iptables is just a configuration tool in the user space.
We know that iptables has four tables and five chains. The four tables are: filter table (for filtering), nat table (for address translation), mangle table (for packet changes), and raw table (to disable connection tracking enabled on the nat table); The five chains are: prerouting (before routing), input (input rules), forward (forwarding rules), output (output rules), and postrouting (after routing).
In this part, we mainly focus on the nat table and its chains. For other parts, if you want to learn, you can refer to iptables-related knowledge. After all, I always remember that I am writing a performance column, not a computer basics column, haha.
From the above information, we can see that there are three ES data services in our cluster, corresponding to three forwarding rules. The match ratios of the first rule is: 0.33333333349; the second rule is: 0.50000000000; and the third rule is 1. These three forwarding rules correspond to the POD IP and port respectively: 10.100.18.197:9200, 10.100.251.67:9200, 10.100.5.5:9200. This means that load balancing can be achieved through these three iptables rules. The diagram below illustrates this:
Let’s assume that there are 30 requests coming in. On ES Data 0, there will be 30x0.33333333349=10 requests; for the remaining 20 requests, there will be 20x0.50000000000=10 requests on ES Data 1; and the last 10 requests will naturally go to ES Data 2. This is a very balanced logic, but I find the ratio in the iptables rules a bit awkward.
Now that we understand this logic, let’s simulate the scenario of querying a product API:
From the data, it frequently happens that a certain node in ES data consumes a high amount of CPU. However, in the global worker monitoring interface we saw earlier, there is no worker with high CPU usage. So here, we need to check the cgroup configuration of ES Data to see what its limitations are.
It means that each POD of ES data is configured with one CPU, no wonder the CPU usage is always high.
Another thing to note is that in our previous examination of the data list, we found that ES Data 0 is on worker-5, ES Data 1 is on worker-7, and ES Data 2 is on worker-9. But now we see that each of them is assigned a separate CPU. In that case, let’s add another CPU and then go back to see the reaction of worker-5/7/9. Why only add one CPU? Because looking at worker-7, the CPU usage is already around 50%, if we add more, I’m afraid it won’t be able to handle it.
Let’s see the result of executing the stress scenario:
It seems… not so good? TPS did not increase.
- Phase 2: Adding Replicas Let’s take a look at the global POD monitoring after adding the CPU:
The high CPU usage is still only on one ES data node, so I want to check the data distribution in ES. Since the load balancing issue has been resolved and we know there are three ES data nodes, we now want to know if each node is being accessed.
pms 0 p 10.100.18.199 _w 32 17690 18363 6.7mb 7820 true true 8.5.1 false
pms 0 p 10.100.18.199 _15 41 2110 0 465.7kb 5500 true true 8.5.1 true
pms 0 p 10.100.18.199 _16 42 21083 30255 9.5mb 5900 true true 8.5.1 false
pms 0 p 10.100.18.199 _17 43 2572 0 568kb 5500 true true 8.5.1 true
pms 0 p 10.100.18.199 _18 44 1403 0 322.9kb 5500 true true 8.5.1 true
pms 0 p 10.100.18.199 _19 45 1856 0 414.1kb 5500 true true 8.5.1 true
pms 0 p 10.100.18.199 _1a 46 1904 0 423kb 5500 true true 8.5.1 true
Why are all the data on one node (10.100.18.199)? It appears to be because there is only one data replica.
green open pms A--6O32bQaSBrJPJltOLHQ 1 0 48618 48618 55.1mb 18.3mb
So, let’s first increase the number of replicas since we have three data nodes. We will add three replicas here:
PUT /pms/_settings
{
"number_of_replicas": 3
}
Let’s again check the data distribution in ES:
pms 0 r 10.100.18.200 _w 32 17690 18363 6.7mb 7820 true true 8.5.1 false
pms 0 r 10.100.18.200 _15 41 2110 0 465.7kb 5500 true true 8.5.1 true
pms 0 r 10.100.18.200 _16 42 21083 30255 9.5mb 5900 true true 8.5.1 false
pms 0 r 10.100.18.200 _17 43 2572 0 568kb 5500 true true 8.5.1 true
pms 0 r 10.100.18.200 _18 44 1403 0 322.9kb 5500 true true 8.5.1 true
pms 0 r 10.100.18.200 _19 45 1856 0 414.1kb 5500 true true 8.5.1 true
pms 0 r 10.100.18.200 _1a 46 1904 0 423kb 5500 true true 8.5.1 true
pms 0 p 10.100.251.69 _w 32 17690 18363 6.7mb 7820 true true 8.5.1 false
pms 0 p 10.100.251.69 _15 41 2110 0 465.7kb 5500 true true 8.5.1 true
pms 0 p 10.100.251.69 _16 42 21083 30255 9.5mb 5900 true true 8.5.1 false
pms 0 p 10.100.251.69 _17 43 2572 0 568kb 5500 true true 8.5.1 true
pms 0 p 10.100.251.69 _18 44 1403 0 322.9kb 5500 true true 8.5.1 true
pms 0 p 10.100.251.69 _19 45 1856 0 414.1kb 5500 true true 8.5.1 true
pms 0 p 10.100.251.69 _1a 46 1904 0 423kb 5500 true true 8.5.1 true
pms 0 r 10.100.140.10 _w 32 17690 18363 6.7mb 7820 true true 8.5.1 false
pms 0 r 10.100.140.10 _15 41 2110 0 465.7kb 5500 true true 8.5.1 true
pms 0 r 10.100.140.10 _16 42 21083 30255 9.5mb 5900 true true 8.5.1 false
pms 0 r 10.100.140.10 _17 43 2572 0 568kb 5500 true true 8.5.1 true
pms 0 r 10.100.140.10 _18 44 1403 0 322.9kb 5500 true true 8.5.1 true
pms 0 r 10.100.140.10 _19 45 1856 0 414.1kb 5500 true true 8.5.1 true
pms 0 r 10.100.140.10 _1a 46 1904 0 423kb 5500 true true 8.5.1 true
We continue to observe the resources of the PODs:
Now, isn’t it happy to see that the CPU of the data node is being utilized?
Let’s take a look at the resources of the worker nodes again:
[root@k8s-master-1 ~]# kubectl get pods -o wide | grep data
elasticsearch-data-0 1/1 Running 0 16m 10.100.18.199 k8s-worker-5 <none> <none>
elasticsearch-data-1 1/1 Running 0 17m 10.100.251.68 k8s-worker-9 <none> <none>
elasticsearch-data-2 1/1 Running 0 18m 10.100.140.9 k8s-worker-2 <none> <none>
Now the ES Data Pods are distributed across worker nodes 2, 5, and 9. Let’s check the global monitoring:
Not bad, the ES data Pods are utilizing the resources quite well. In fact, here we can further increase the CPU, as ES is known for consuming a lot of CPU and memory. However, when we initially configured it, we didn’t consider this.
Finally, let’s see the effect of the optimization:
Oh my, the TPS quickly reached around 900! This optimization result is very good.
Now, looking back at the first stage, increasing the CPU did not have an effect mainly because there were too few replicas. In fact, there are many other details to optimize in ES. However, in this course, I want to provide you with a holistic analysis approach and logic, rather than focusing on each individual parameter. Therefore, we will not discuss specific parameter adjustments here if you want to do further optimizations on ES. After analyzing the business, you can determine the architecture, data indexing, shard information of ES, and then design a reasonable ES deployment.
Summary #
In this class, we’ve seen that APM tools can also have limitations. So, when we analyze a specific component and want to go further, we have to rely on our own technical skills.
When we encounter requests imbalance, we must first check if there is a problem with the load balancing logic. When we noticed that the ES client was imbalanced, we looked at the principles of iptables. After finding that there was only one forwarding rule in iptables, the next step was to refresh the forwarding rule.
After balancing the ES client forwarding, we noticed high CPU usage on the ES data single node. Since ES data is in the POD, we naturally thought of checking the cgroup limitations.
After adding CPU, we found that TPS did not increase. At this point, we need to look at the logic of ES. The strength of ES lies in its ability to handle queries with multiple replicas and shards, so when we added replicas, CPU usage increased and TPS naturally improved.
After a series of actions, we finally utilized the resources. This is also what I have been emphasizing: the goal of the first stage of performance optimization is to utilize resources and then consider more detailed optimizations.
Homework #
Finally, I have two questions for you to consider:
- What is the main direction of analysis when the load is unbalanced?
- When do you need to look at the internal implementation logic of a component?
Remember to discuss and exchange your thoughts with me in the comments section. Every thought will help you make further progress.
If you have gained something from reading this article, feel free to share it with your friends for mutual learning and progress. See you in the next lesson!