39 Case Study How to Mitigate Performance Downturn Caused by Ddo S Attacks

39 Case Study How to Mitigate Performance Downturn Caused by DDoS Attacks #

Hello, I’m Ni Pengfei.

In the previous section, I taught you how to use tcpdump and Wireshark, and I used several case studies to show you how to use these two tools to analyze the network transmission process. When encountering network performance issues, don’t forget that you can use tcpdump and Wireshark to capture the actual transmitted network packets and investigate potential performance issues.

Today, let’s take a look at another issue: how to mitigate the performance degradation caused by DDoS (Distributed Denial of Service) attacks.

Introduction to DDoS #

The precursor to DDoS is DoS (Denial of Service), which refers to the use of a large number of legitimate requests to consume excessive target resources, thereby rendering the target service unable to respond to normal requests.

DDoS (Distributed Denial of Service) builds on DoS by employing a distributed architecture to launch simultaneous attacks on a target host using multiple host machines. Even if the target service has deployed network defense devices, it is still powerless against a large volume of network requests.

For example, the largest-known volumetric attack to date was the DDoS attack suffered by GitHub last year. The peak traffic reached 1.35 Tbps, and the PPS (packets per second) exceeded 126.9 million.

From an attack principle perspective, DDoS can be classified into the following types.

The first type is bandwidth exhaustion. Both servers and network devices such as routers and switches have a fixed upper limit on bandwidth. When the bandwidth is exhausted, network congestion occurs, preventing the transmission of other normal network packets.

The second type is resource exhaustion of the operating system. The normal operation of network services requires certain system resources, such as CPU, memory, and software resources like connection tables. Once these resources are exhausted, the system cannot process other normal network connections.

The third type is the consumption of running resources of an application. Running an application usually requires interacting with other resources or systems. If the application is continuously busy handling invalid requests, it will slow down the processing of normal requests or even fail to respond.

For example, constructing a large number of different domain names to attack DNS servers leads to DNS servers constantly executing iterative queries and updating their caches. This greatly consumes the resources of the DNS server, resulting in slower DNS responses.

Regardless of the type of DDoS attack, the harm is enormous. So, how can we detect if a system is under a DDoS attack and how should we respond to such attacks? Next, let’s explore these questions together through a case study.

Case Preparation #

The following case is still based on Ubuntu 18.04 and is also applicable to other Linux systems. The environment used for this case is as follows:

Machine configuration: 2 CPUs, 8GB memory.
Pre-install docker, sar, hping3, tcpdump, curl, and other tools, such as apt-get install docker.io hping3 tcpdump curl.

You should be quite familiar with these tools. Among them, hping3, which has been introduced in the case of increased CPU usage caused by system soft interrupts, can construct TCP/IP protocol packets to perform security auditing, firewall testing, and DoS attack testing on the system.

This case requires three virtual machines, and I have drawn a diagram to show their relationship.

As you can see, one of the virtual machines runs Nginx and is used to simulate the web server to be analyzed, while the other two are used as clients of the web server, with one being used for DoS attacks and the other being a normal client. The use of multiple virtual machines is naturally for mutual isolation and to avoid “cross-infection”.

Since only one machine is used as the source of the attack in this case, the attack is actually a traditional DoS attack, rather than a DDoS attack.

Next, we open three terminals and SSH login to the three machines separately (the following steps assume that the terminal number is consistent with the VM number in the diagram), and install the mentioned tools.

As in the previous cases, all the following commands are assumed to be run as the root user by default. If you are logged in to the system as a regular user, run the command sudo su root to switch to the root user.

Next, we will enter the case operation phase.

Case Study #

First, in the first terminal, execute the following command to run the case, which is to start a basic Nginx application:

# Run Nginx service and open port 80 to the outside
# --network=host means use host network (for easier troubleshooting later)
$ docker run -itd --name=nginx --network=host nginx

Then, in the second and third terminals, use curl to access the port that Nginx is listening on to confirm that Nginx is started successfully. Assuming 192.168.0.30 is the IP address of the virtual machine where Nginx is located, you should see the following output after running the curl command:

# -w shows only the HTTP status code and total time, -o redirects the response to /dev/null
$ curl -s -w 'Http code: %{http_code}\nTotal time:%{time_total}s\n' -o /dev/null http://192.168.0.30/
...
Http code: 200
Total time: 0.002s

From here, you can see that under normal circumstances, accessing Nginx only takes 2ms (0.002s).

Next, in the second terminal, run the hping3 command to simulate a DoS attack:

# -S sets the SYN (synchronization) sequence number for the TCP protocol, -p sets the destination port to 80
# -i u10 sends a network frame every 10 microseconds
$ hping3 -S -p 80 -i u10 192.168.0.30

Now, go back to the first terminal, and you will notice that everything is much slower no matter what command you execute. However, when practicing, please note:

If the issue is not very obvious, try reducing the value of u10 in the parameters (such as changing it to u1), or add the –flood option.
If your first terminal becomes completely unresponsive, adjust u10 appropriately (such as increasing it to u30), otherwise you won’t be able to operate VM1 through SSH later.

Then, in the third terminal, execute the following command to simulate a normal client connection:

# --connect-timeout sets the connection timeout
$ curl -w 'Http code: %{http_code}\nTotal time:%{time_total}s\n' -o /dev/null --connect-timeout 10 http://192.168.0.30
...
Http code: 000
Total time: 10.001s
curl: (28) Connection timed out after 10000 milliseconds

You can see that in the third terminal, the normal client’s connection times out and does not receive a response from the Nginx service.

What is the problem? Let’s go back to the first terminal and check the network status. You should remember sar, which we have used multiple times before. It can be used to observe PPS (packets per second) as well as BPS (bytes per second).

We can go back to the first terminal and execute the following command:

$ sar -n DEV 1
08:55:49        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
08:55:50      docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
08:55:50         eth0  22274.00    629.00   1174.64     37.78      0.00      0.00      0.00      0.02
08:55:50           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

The meaning of each column in the sar output was explained in the previous section on Linux networking basics. You can click here to view it or execute man sar to query the manual.

From the output of sar this time, you can see that the received PPS has already reached over 20000, but the BPS is only 1174 kB, which means the size of each packet is only 54B (1174*1024/22274=54).

This is obviously a small packet. But what kind of packet is it? Let’s use tcpdump to capture and analyze it.

In the first terminal, execute the following tcpdump command:

# -i eth0 captures only on the eth0 network interface, -n does not resolve protocol names and hostnames
# tcp port 80 captures only TCP protocol frames with port 80
$ tcpdump -i eth0 -n tcp port 80
09:15:48.287047 IP 192.168.0.2.27095 > 192.168.0.30: Flags [S], seq 1288268370, win 512, length 0
09:15:48.287050 IP 192.168.0.2.27131 > 192.168.0.30: Flags [S], seq 2084255254, win 512, length 0
09:15:48.287052 IP 192.168.0.2.27116 > 192.168.0.30: Flags [S], seq 677393791, win 512, length 0
09:15:48.287055 IP 192.168.0.2.27141 > 192.168.0.30: Flags [S], seq 1276451587, win 512, length 0
09:15:48.287068 IP 192.168.0.2.27154 > 192.168.0.30: Flags [S], seq 1851495339, win 512, length 0
...

In this output, Flags [S] indicates that these are SYN packets. A large number of SYN packets indicate a SYN Flood attack. If you use Wireshark mentioned in the previous section to observe, you can see the SYN Flood process more intuitively:

In fact, SYN Flood is the most classic DDoS attack method on the Internet. From this figure, you can also see its working principle:

The client constructs a large number of SYN packets to request TCP connections.
When the server receives the packets, it will send SYN+ACK packets to the source IP and wait for the final ACK packet of the three-way handshake until timeout.

This kind of TCP connection in the waiting state is usually referred to as a half-open connection. Due to the limited size of the connection table, a large number of half-open connections can quickly fill up the connection table, making it impossible to establish new TCP connections.

Refer to the following TCP state diagram, and you can see that the server-side TCP connection is in the SYN_RECEIVED state at this time:

(Image source: Wikipedia)

This actually suggests that the key to viewing half-open connections lies in the connections in the SYN_RECEIVED state. We can use netstat to check the status of all connections, but note that the SYN_RECEIVED state is usually abbreviated as SYN_RECV. We continue in Terminal 1 and execute the following netstat command:

# -n means do not resolve names, -p means display the process of the connection
$ netstat -n -p | grep SYN_REC
tcp        0      0 192.168.0.30:80          192.168.0.2:12503      SYN_RECV    -
tcp        0      0 192.168.0.30:80          192.168.0.2:13502      SYN_RECV    -
tcp        0      0 192.168.0.30:80          192.168.0.2:15256      SYN_RECV    -
tcp        0      0 192.168.0.30:80          192.168.0.2:18117      SYN_RECV    -
...

From the results, you can see that there are a large number of connections in SYN_RECV state, with the source IP address being 192.168.0.2.

Furthermore, we can use the wc tool to count the total number of SYN_RECV connections:

$ netstat -n -p | grep SYN_REC | wc -l
193

After identifying the source IP, we can solve the SYN attack problem by simply dropping the related packets. For this, iptables can assist you. You can execute the following iptables command in Terminal 1:

$ iptables -I INPUT -s 192.168.0.2 -p tcp -j REJECT

Then, go back to Terminal 3 and execute the curl command again to check the normal user’s access to Nginx:

$ curl -w 'Http code: %{http_code}\nTotal time:%{time_total}s\n' -o /dev/null --connect-timeout 10 http://192.168.0.30
Http code: 200
Total time:1.572171s

Now, you will notice that normal users can access Nginx, but the response is slower, increasing from the previous 2ms to the current 1.5s.

However, usually the source IP in a SYN Flood attack is not fixed. For example, you can use the –rand-source option in the hping3 command to randomize the source IP. In this case, the previous method will not work.

Thankfully, there are many other methods to achieve similar goals. For example, you can use the following two approaches to limit the rate of SYN packets:

# Limit the number of SYN packets to 1 per second
$ iptables -A INPUT -p tcp --syn -m limit --limit 1/s -j ACCEPT

# Limit the number of connections established by a single IP to 10 within 60 seconds
$ iptables -I INPUT -p tcp --dport 80 --syn -m recent --name SYN_FLOOD --update --seconds 60 --hitcount 10 -j REJECT

With these steps, we have taken preliminary measures to mitigate SYN Flood attacks. However, this may not be enough because our example only involves a single attacker.

If multiple machines simultaneously send SYN Flood attacks, these methods may be ineffective. It is possible that you will be unable to SSH into the machine (SSH also relies on TCP) and execute the aforementioned troubleshooting commands.

Therefore, additional TCP optimizations need to be performed on the system.

For instance, SYN Floods lead to a sharp increase in the number of connections in SYN_RECV state. As you can see in the netstat command above, there are over 190 connections in this half-open state.

However, the number of half-open connections is limited. By executing the following command, you can check that the default half-open connection capacity is only 256:

$ sysctl net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_max_syn_backlog = 256

In other words, once the number of SYN packets slightly increases, you will be unable to SSH into the machine. Therefore, you should increase the capacity for half-open connections. For example, you can use the following command to set it to 1024:

$ sysctl -w net.ipv4.tcp_max_syn_backlog=1024
net.ipv4.tcp_max_syn_backlog = 1024

Additionally, when each SYN_RECV connection fails, the kernel will automatically retry, and the default retry count is 5. You can reduce this count to 1 by executing the following command:

$ sysctl -w net.ipv4.tcp_synack_retries=1
net.ipv4.tcp_synack_retries = 1

Besides these optimizations, TCP SYN Cookies are another method specifically designed to defend against SYN Flood attacks. SYN Cookies calculate a hash value (SHA1) based on connection information such as source address, source port, destination address, destination port, and an encryption seed (e.g., system startup time). This hash value is used as a cookie, acting as a sequence number to respond to SYN+ACK packets and releasing the connection state. When the client sends the final ACK after completing the three-way handshake, the server calculates this hash value again to confirm that it matches the SYN+ACK packet returned last time before entering the TCP connection state.

Therefore, once SYN Cookies are enabled, there is no need to maintain half-open connection states, which eliminates the limit on the number of half-open connections.

To enable TCP SYN Cookies, you can execute the following command:

$ sysctl -w net.ipv4.tcp_syncookies=1
net.ipv4.tcp_syncookies = 1

Note that when TCP syncookies are enabled, the kernel option net.ipv4.tcp_max_syn_backlog becomes ineffective.

To ensure the persistence of these configurations, you should also write them to the /etc/sysctl.conf file. For example:

$ cat /etc/sysctl.conf
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_max_syn_backlog = 1024

However, remember that the configurations written to /etc/sysctl.conf only take effect after executing the sysctl -p command.

Lastly, after finishing the example, don’t forget to execute the docker rm -f nginx command to clean up the previously started Nginx application.

How to Defend Against DDoS Attacks #

Here we come to the end of today’s case study. However, you must still have questions. You may have noticed that today’s topic is “mitigation” rather than “solving” the DDoS problem.

Why is it about mitigating DDoS instead of solving it? Moreover, the methods mentioned in today’s case study only prevent Nginx service from timing out, but the latency is still much higher than the initial 2ms.

In fact, when DDoS packets reach the server, the mechanisms provided by Linux can only mitigate the attacks rather than completely resolve them. Even small packet attacks like SYN Flood can consume a large amount of resources in the Linux kernel, leading to slow processing of other network packets.

Although you can adjust kernel parameters to mitigate the performance issues caused by DDoS, you still can’t completely solve it, just like the case study.

In the previous articles on C10K, C100K, I also mentioned that the lengthy protocol stack in the Linux kernel becomes a tremendous burden when the PPS (packets per second) is high. The same goes for DDoS attacks.

Therefore, the methods mentioned in the C10M article are also applicable here. For example, you can build a DDoS solution based on XDP or DPDK to identify and discard DDoS traffic before or bypassing the kernel network protocol stack, thereby avoiding the consumption of system resources by DDoS attacks.

However, for traffic-based DDoS attacks, when the server’s bandwidth is exhausted, there is nothing that can be done internally. At this point, you can only rely on external network devices to identify and block the traffic (provided that the network devices can withstand the traffic attacks). For example, you can purchase professional intrusion detection and defense devices, and configure traffic scrubbing devices to block malicious traffic.

Since DDoS attacks are challenging to defend against, does it mean that Linux servers don’t focus on this aspect and leave it all to professional network devices?

Of course not, because DDoS attacks are not necessarily caused by high traffic or PPS. Sometimes, slow requests can also cause significant performance degradation (known as slow-speed DDoS).

For example, many attacks targeting applications disguise themselves as normal users requesting resources. In this case, the request traffic itself may not be significant, but the response traffic may be large, and the application may also consume a large amount of resources to process it.

In such cases, the application needs to consider identifying and rejecting such malicious traffic as early as possible, such as by using caching effectively, adding a Web Application Firewall (WAF), and utilizing CDN (Content Delivery Network), among others.

Summary #

Today, we have learned about mitigation methods for Distributed Denial of Service (DDoS) attacks. DDoS attacks exploit a large number of forged requests to consume a significant amount of resources, making it difficult for the target service to respond to legitimate user requests.

Due to the distributed, high traffic, and difficult-to-trace nature of DDoS attacks, there is currently no method that can completely defend against the problems caused by DDoS attacks. The only option is to mitigate the impact.

For example, you can purchase professional traffic cleaning devices and network firewalls to block malicious traffic at the network entrance, allowing only legitimate traffic to enter the servers in the data center.

In Linux servers, you can increase the server’s ability to withstand attacks and reduce the impact of DDoS attacks on normal services through kernel tuning, DPDK, XDP, and other methods. In applications, you can use various levels of caching, WAF, CDN, and other methods to mitigate the impact of DDoS attacks on the application.

Reflection #

Finally, I’ll leave you with a thought-provoking question.

After reading today’s case, you may find it familiar. In fact, it is an extension of the case of high CPU usage caused by system soft interrupts. At that time, we analyzed it from the perspective of soft interrupt CPU usage. In other words, DDoS attacks can cause a rise in soft interrupt CPU usage.

Recall the case and analysis approach back then, combined with today’s case, do you think there is a better method to detect DDoS attacks? Besides tcpdump, what other methods can be used to find the source addresses of these attacks?

Feel free to discuss with me in the comments and share this article with your colleagues and friends. Let’s practice in real scenarios and make progress through communication.