47 Case Study the Server Constantly Drops Packets, What Should I Do Part One

47 Case Study The Server Constantly Drops Packets, What Should I Do Part One #

Hello, I’m Ni Pengfei.

In the previous section, we discussed the analysis methods for performance degradation after application containerization. Let’s briefly review together.

Containers use the namespace technology provided by the Linux kernel to isolate the execution of different applications and manage the application’s dependency environment with a unified image. This brings great convenience in application management and maintenance and further promotes new generation technologies such as microservices and cloud-native architectures.

However, although there are many advantages, containerization can also have certain impacts on application performance. For example, as we analyzed in the previous section, Java applications are prone to issues such as slow startup and Out of Memory (OOM) exits after running for a period of time. When you encounter such problems, don’t panic, the various approaches we discussed in the four fundamental modules are still applicable.

In fact, many of the case studies in our column are running inside containers. After containerization, applications are isolated through namespaces. So, when analyzing, don’t forget to consider the comprehensive analysis of namespaces, cgroups, iptables, and so on. For example:

  • cgroups affect the execution of containerized applications;

  • NAT in iptables affects the network performance of containers;

  • Overlay filesystems affect the I/O performance of applications, and so on.

Regarding the impact of NAT, I have already introduced many optimization methods in the Optimizing NAT Performance article in the network module. Today, let’s look at another scenario, which is the analysis method for packet loss.

Packet loss refers to the situation where data packets are discarded during the process of data transmission in the network, due to various reasons, before they reach the application program. The number of discarded packets divided by the total number of transmitted packets is commonly known as the packet loss rate. The packet loss rate is one of the most critical indicators in network performance.

Packet loss usually leads to significant performance degradation, especially for TCP, where packet loss often indicates network congestion and retransmission, leading to increased network latency and reduced throughput.

Next, I will take the widely used reverse proxy server Nginx as an example to show you how to analyze network packet loss issues. Due to the amount of content, this case study will be divided into two parts. Today, let’s first look at the first part.

Preparation #

Today’s case requires the use of two virtual machines, both based on Ubuntu 18.04, which is also applicable to other Linux systems. The environment I used for the case is as follows:

  • Machine configuration: 2 CPUs, 8GB memory.

  • Pre-install tools such as docker, curl, hping3, etc., using the command apt install docker.io curl hping3.

We have used these tools multiple times in previous cases, so I will not repeat the introduction here.

Now, open two terminals, log in to these two virtual machines respectively, and install the mentioned tools.

Note: All the following commands are run by default as the root user. If you log in to the system as a normal user, please run the command sudo su root to switch to the root user.

If there are any problems during the installation process, you can search for solutions online. If you are unable to solve them, remember to ask me in the comments section.

With that, the preparation work is complete. Next, let’s officially move on to the operation section.

Case Study #

The case we are going to analyze today is an Nginx application, as shown in the figure below. hping3 and curl are the clients for Nginx.

To facilitate your running, I have packaged it into a Docker image and pushed it to Docker Hub. You can directly follow the steps below to run it.

Execute the following command in Terminal 1 to start the Nginx application and listen on port 80. If everything is normal, you should see the following output:

$ docker run --name nginx --hostname nginx --privileged -p 80:80 -itd feisky/nginx:drop
dae0202cc27e5082b282a6aeeb1398fcec423c642e63322da2a97b9ebd7538e0

Then, execute the command docker ps to check the status of the container. You will find that the container is already in running state (Up):

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                NAMES
dae0202cc27e        feisky/nginx:drop   "/start.sh"         4 minutes ago       Up 4 minutes        0.0.0.0:80->80/tcp   nginx

However, from the output of docker ps, we can only know that the container is in running state. As for whether Nginx can handle external requests normally, further confirmation is needed.

Next, switch to Terminal 2 and execute the following hping3 command to further verify whether Nginx can be accessed normally. Note that here I did not use ping because ping is based on the ICMP protocol, while Nginx uses the TCP protocol.

# -c indicates sending 10 requests, -S indicates using TCP SYN, -p specifies the port as 80
$ hping3 -c 10 -S -p 80 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=3 win=5120 rtt=7.5 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=4 win=5120 rtt=7.4 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=5 win=5120 rtt=3.3 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=7 win=5120 rtt=3.0 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=6 win=5120 rtt=3027.2 ms

--- 192.168.0.30 hping statistic ---
10 packets transmitted, 5 packets received, 50% packet loss
round-trip min/avg/max = 3.0/609.7/3027.2 ms

From the output of hping3, we can see that although 10 request packets were sent, only 5 replies were received, with 50% packet loss. By observing the RTT of each request, we can find that there is also a large fluctuation in RTT, with a small value of only 3ms and a large value of 3s.

Based on this output, we can basically conclude that packet loss has occurred. It can be speculated that the 3s RTT is most likely due to retransmission after packet loss. So where exactly is the packet loss?

Before troubleshooting, let’s recall the network transmission and reception process on Linux. Let’s analyze theoretically where packet loss might occur. You may take out a pen and paper and meditate while organizing your thoughts on paper before continuing with the following content.

Here, to help you understand the principle of network packet loss, I have drawn a diagram. You can save and print it for reference:

From the diagram, you can see that the positions where packet loss may occur actually cover the entire network protocol stack. In other words, there is a possibility of packet loss throughout the process. For example, starting from the bottom:

  • Between the two VMs, transmission failures may occur, such as network congestion, line errors, etc.
  • After receiving packets, the circular buffer in the network card may overflow and drop packets.
  • At the link layer, packets may be dropped due to network frame checksum failure, QoS, etc.
  • At the IP layer, packets may be dropped due to routing failure or packet size exceeding MTU.
  • At the transport layer, packets may be dropped due to ports not being listened to, resource occupation exceeding kernel limits, etc.
  • At the socket layer, packets may be dropped due to socket buffer overflow.
  • At the application layer, packets may be dropped due to abnormal application behavior.
  • In addition, if iptables rules are configured, these network packets may also be dropped due to iptables filtering rules.

Of course, the above problems may occur simultaneously on both machines involved in communication. However, since we haven’t made any modifications to VM2 and VM2 is only running a simple hping3 command, let’s assume it is problem-free.

To simplify the entire troubleshooting process, we can further assume that VM1’s network and kernel configurations are problem-free. In this case, the potential problem locations would all be within the container.

Now let’s switch back to terminal 1 and execute the following command to enter the container’s terminal:

$ docker exec -it nginx bash
root@nginx:/#

Here, let me briefly explain that all the following analyses, prefixed with _root@nginx:/#_, are operations performed within the container.

Note: In the actual environment, problems can occur inside and outside the container. However, don’t worry, the steps and approach for analyzing inside and outside the container are the same, just that it may take more time.

Now, let’s systematically troubleshoot the packet dropping issue by examining each layer of the protocol stack.

First, let’s take a look at the lowest layer, the link layer. When packet loss occurs due to buffer overflow or other reasons, Linux records the number of receive and transmit errors in the network card’s statistics.

You can use ethtool or netstat to view the packet loss records of the network card. For example, you can execute the following command within the container to check the packet loss situation:

root@nginx:/# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       100       31      0      0 0             8      0      0      0 BMRU
lo       65536        0      0      0 0             0      0      0      0 LRU

In the output, RX-OK, RX-ERR, RX-DRP, RX-OVR represent the total number of packets received, total number of errors, number of dropped packets due to other reasons (such as insufficient memory) after entering the ring buffer, and number of dropped packets due to ring buffer overflow, respectively.

Similarly, TX-OK, TX-ERR, TX-DRP, TX-OVR represent similar metrics for transmissions, with the corresponding meanings.

Note that due to the virtual network card of Docker container, it is actually a pair of veth interfaces, with one end connected to eth0 in the container and the other end connected to the docker0 bridge in the host. The veth driver does not implement network statistics, so the command ethtool -S cannot provide the summary information of the data transmitted and received by the network card.

From this output, we did not find any errors, indicating that there are no dropped packets on the virtual network card of the container. However, please note that if QoS is configured using tools such as tc, the packet loss caused by tc rules will not be included in the statistics of the network card.

Therefore, next, we need to check whether tc rules are configured on eth0 and check if there are any dropped packets. Let’s continue in the container terminal and execute the following tc command, but this time, we need to add the -s option to output statistics:

root@nginx:/# tc -s qdisc show dev eth0
qdisc netem 800d: root refcnt 2 limit 1000 loss 30%
 Sent 432 bytes 8 pkt (dropped 4, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

From the output of tc, we can see that a network emulation queuing discipline (qdisc netem) is configured on eth0, and the packet loss rate is set to 30% (loss 30%). Looking at the statistics, 8 packets were sent, but 4 were dropped.

It seems that this is where the response packets from Nginx are being dropped by the netem module.

Now that the problem has been found, the solution is simple, just delete the netem module. We can continue in the container terminal and execute the following command to remove the netem module from tc:

root@nginx:/# tc qdisc del dev eth0 root netem loss 30%

Has the problem been resolved after removal? Let’s switch to Terminal 2 and re-execute the previous hping3 command to see if the problem still exists:

$ hping3 -c 10 -S -p 80 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=0 win=5120 rtt=7.9 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=2 win=5120 rtt=1003.8 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=5 win=5120 rtt=7.6 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=6 win=5120 rtt=7.4 ms
len=44 ip=192.168.0.30 ttl=63 DF id=0 sport=80 flags=SA seq=9 win=5120 rtt=3.0 ms

--- 192.168.0.30 hping statistic ---
10 packets transmitted, 5 packets received, 50% packet loss
round-trip min/avg/max = 3.0/205.9/1003.8 ms

Unfortunately, from the output of hping3, we can see that the packet loss is still 50% and the RTT fluctuates greatly, ranging from 3 ms to 1 second.

Obviously, the problem has not been resolved and packet loss is still occurring. However, since the link layer has been investigated, we will continue to analyze the higher layers and see if there are any issues at the network layer and transport layer.

Network Layer and Transport Layer #

We know that there are many factors that can cause packet loss at the network layer and transport layer. However, it is actually quite simple to confirm packet loss because Linux already provides us with the summary of sent and received packets for each protocol.

Let’s continue in the container terminal and execute the netstat -s command to see the summary of protocols, including received and sent packets, as well as error information:

root@nginx:/# netstat -s
Ip:
    Forwarding: 1
    31 total packets received
    0 forwarded
    0 incoming packets discarded
    25 incoming packets delivered
    15 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed
    ICMP input histogram:
    0 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
Tcp:
    0 active connection openings
    0 passive connection openings
    11 failed connection attempts
    0 connection resets received
    0 connections established
    25 segments received
    21 segments sent out
    4 segments retransmitted
    0 bad segments received
    0 resets sent
Udp:
    0 packets received
    ...
TcpExt:
    11 resets received for embryonic SYN_RECV sockets
    0 packet headers predicted
    TCPTimeouts: 7
    TCPSynRetrans: 4
    ...

netstat summarizes the received and sent statistics for various protocols such as IP, ICMP, TCP, and UDP. However, since we are investigating packet loss, we mainly focus on the error count, packet loss count, and retransmission count.

According to the above output, you can see that only the TCP protocol has packet loss and retransmission, namely:

  • 11 failed connection attempts
  • 4 segments retransmitted
  • 11 resets received for embryonic SYN_RECV sockets
  • 4 SYN retransmissions
  • 7 timeouts

This result tells us that the TCP protocol experienced multiple timeouts and failed connection attempts, and the main error was the reset of half-open connections. In other words, the main failures were due to three-way handshake failures.

However, although we see many failures here, the specific root cause of the failures cannot be determined. Therefore, we still need to continue the analysis along the protocol stack. How should we analyze the next layers? You can take a moment to think and explore on your own, and in the next section, we will continue to discuss together.

Summary #

Packet loss on the network usually leads to significant performance degradation, especially for TCP. Packet loss usually indicates network congestion and retransmission, which further increases network latency and reduces throughput.

In today’s case, we learned how to analyze network packet loss starting from the link layer, network layer, and transport layer. However, we have not yet identified the ultimate performance bottleneck at the end of the case. In the next section, I will continue to explain it to you.

Reflection #

Finally, I have a question for you to think about, which is also the question raised at the end of the case.

Today we analyzed the link layer, network layer, and transport layer, etc. However, according to the TCP/IP protocol stack and Linux network sending and receiving principles, there are still many areas that we haven’t analyzed. So, how should we continue our analysis to crack this case and find the “real culprit”?

Feel free to discuss with me in the comment section, and also feel free to share this article with your colleagues and friends. Let’s practice in real scenarios and improve through communication together.