09 Fundamentals Understanding Linux Soft Lockup

09 Fundamentals Understanding Linux Soft Lockup #

Hello, I am Ni Pengfei.

In the previous lesson, I took you through the analysis method of iowait (which is the CPU usage rate while waiting for I/O) with an example of an uninterruptible process. Remember, the uninterruptible state of a process is a system protection mechanism that ensures the hardware interaction process is not accidentally interrupted. Therefore, it is normal for a process to be in an uninterruptible state for a short period of time.

However, when a process stays in an uninterruptible state for a long time, you need to be careful. In this case, you can use tools like dstat, pidstat, etc. to confirm if it is an issue with disk I/O and then investigate the related processes and disk devices. As for the performance issues with disk I/O, you don’t need to memorize them specifically for now. I will explain it in detail in the subsequent I/O section, and once you understand it, you will remember it.

In addition to iowait, high usage of softirq (software interrupt) CPU is also a common performance issue. In the next two lessons, we will learn about softirq, and I will use the case of the most common reverse proxy server, Nginx, to analyze this situation with you.

Understanding Interrupts through “Ordering Food Delivery” #

When it comes to interrupts, I briefly explained their meaning in the earlier article on “context switching”. Let’s have a quick review. Interrupts are a mechanism that the system uses to respond to hardware device requests. They interrupt the normal scheduling and execution of processes, and then call interrupt handlers in the kernel to respond to device requests.

You may wonder why we need interrupts. Let me give you an example from daily life to help you appreciate the charm of interrupts.

Let’s say you ordered some food delivery, but you are unsure when it will arrive and have no other way to check its progress. However, delivery drivers don’t wait for people. If there’s no one to receive the delivery when they arrive, they simply leave. So, you are left waiting anxiously, occasionally going to the door to check if the delivery has arrived, unable to do anything else.

However, if you arrange with the delivery driver to give you a call after the delivery arrives, then you don’t have to wait anxiously. You can go about doing other things until the phone rings, at which point you can answer the call and retrieve your food.

In this case, the “phone call” is actually an interrupt. When you haven’t received the call, you can do other things; only after receiving the call (i.e., when the interrupt occurs), do you perform another action: retrieving the food.

From this example, you can see that interrupts are actually an asynchronous event handling mechanism, which can improve the system’s ability to handle concurrency.

Since interrupt handling routines interrupt the execution of other processes, in order to minimize the impact on the scheduling of normal processes, interrupt handlers need to run as quickly as possible. If the tasks to be performed by an interrupt are not too complex, there won’t be a big problem in handling them. However, if an interrupt has a lot of tasks to handle, the interrupt service routine may take a long time to run.

In particular, when an interrupt is being serviced, it temporarily disables other interrupts. This means that until the previous interrupt handling is completed, other interrupts cannot be responded to, which means interrupts may be lost.

Let’s continue with the example of ordering food delivery. Suppose you ordered 2 items: a main course and a drink, and they will be delivered by 2 different drivers. This time, you don’t have to wait all the time because you have arranged to retrieve both items by phone. But here’s a problem.

When the first delivery arrives, the driver calls you and talks for a long time about how to handle the invoice. At the same time, the second driver also arrives and wants to call you.

Clearly, because the line is busy (since interrupts are disabled), the second driver’s call cannot get through. So, the second driver will probably try a few times and then leave (resulting in a lost interrupt).

Soft Interrupt #

If you understand the concept of “ordering takeaway”, then it will be easy to understand the system’s interrupt mechanism. In fact, in order to solve the problems of long interrupt processing time and interrupt loss, Linux divides the interrupt handling process into two phases, namely the top half and the bottom half:

The top half is used to quickly handle interrupts, it runs in interrupt-disabled mode and is mainly responsible for handling tasks that are closely related to hardware or time-sensitive.
The bottom half is used to defer the completion of unfinished tasks in the top half, usually running as kernel threads.

For example, in the previous example of ordering takeaway, the top half is when you answer the phone and inform the delivery person that you are aware of the situation and will discuss everything later when you meet. After that, the phone can be hung up. The bottom half is actually the action of ordering takeaway and the later discussion on handling the invoice.

This way, the first delivery person will not take up too much of your time, and when the second delivery person arrives, you can still make and receive calls normally.

Besides ordering takeaway, let me give you another example of the most common case of receiving network packets by a network card, to help you understand it better.

When the network card receives a packet, it will notify the kernel that new data has arrived through a hardware interrupt. At this time, the kernel should call the interrupt handler to respond to it. Take a moment to think about what the top half and the bottom half are responsible for in this case.

For the top half, since it needs to be processed quickly, it actually reads the data from the network card into memory, updates the status of the hardware registers (indicating that the data has been read), and then sends a soft interrupt signal to notify the bottom half to perform further processing.

When the bottom half is awakened by the soft interrupt signal, it needs to find the network data from memory, and then parse and process the data layer by layer according to the network protocol stack until it is delivered to the application program.

Therefore, these two phases can be understood as follows:

The top half directly handles hardware requests, which are commonly referred to as hard interrupts and are characterized by quick execution;
The bottom half is triggered by the kernel, which is commonly referred to as soft interrupts and is characterized by delayed execution.

In practice, the top half interrupts the task that the CPU is currently executing, and then immediately executes the interrupt handler. The bottom half is executed as a kernel thread, and each CPU corresponds to a soft interrupt kernel thread named “ksoftirqd/CPU number”. For example, the name of the soft interrupt kernel thread corresponding to CPU 0 is ksoftirqd/0.

However, it is important to note that soft interrupts not only include the bottom half of the interrupt handling program of hardware devices mentioned earlier, but also some kernel-defined events, such as kernel scheduling and RCU lock (short for Read-Copy Update, RCU is one of the most commonly used locks in the Linux kernel).

So, how do you know which soft interrupts exist in your system?

Viewing Soft Interrupts and Kernel Threads #

Do you remember the proc file system we mentioned earlier? It is a mechanism for communication between the kernel space and user space, which can be used to view kernel data structures or dynamically modify kernel configurations. Among them:

/proc/softirqs provides information about the running status of soft interrupts;
/proc/interrupts provides information about the running status of hardware interrupts.

By running the following command and checking the contents of the /proc/softirqs file, you can see the accumulated number of soft interrupt runs for different types of soft interrupts on different CPUs:

```
$ cat /proc/softirqs
                    CPU0       CPU1
          HI:          0          0
       TIMER:     811613    1972736
      NET_TX:         49          7
      NET_RX:    1136736    1506885
       BLOCK:          0          0
    IRQ_POLL:          0          0
     TASKLET:     304787       3691
       SCHED:     689718    1897539
     HRTIMER:          0          0
         RCU:    1330771    1354737
```

When viewing the contents of the /proc/softirqs file, you should pay special attention to the following two points.

First, pay attention to the types of soft interrupts, which are the content in the first column of this interface. From the first column, you can see that there are 10 categories of soft interrupts, each corresponding to a different type of work. For example, NET_RX represents network receive interrupts, while NET_TX represents network transmit interrupts.

Second, pay attention to the distribution of the same type of soft interrupt on different CPUs, which is the content in the same row. Normally, the cumulative number of the same interrupt on different CPUs should be similar. For example, in this interface, the number of interrupts for NET_RX on CPU0 and CPU1 is basically in the same order of magnitude, with little difference.

However, you may notice that the distribution of TASKLET on different CPUs is not even. TASKLET is the most commonly used soft interrupt implementation mechanism, and each TASKLET only runs once and then ends, running only on the CPU where the function calling it is located.

Therefore, using TASKLET is particularly convenient, but it may also have some problems, such as scheduling imbalance caused by running on only one CPU, and performance limitations caused by not being able to run in parallel on multiple CPUs.

In addition, as mentioned earlier, soft interrupts actually run as kernel threads, and each CPU corresponds to a soft interrupt kernel thread named ksoftirqd/CPU number. So how can you view the running status of these threads?

In fact, you can use the ps command to do this, for example, execute the following command:

```
$ ps aux | grep softirq
root         7  0.0  0.0      0     0 ?        S    Oct10   0:01 [ksoftirqd/0]
root        16  0.0  0.0      0     0 ?        S    Oct10   0:01 [ksoftirqd/1]
```

Note that the names of these threads are enclosed in square brackets, indicating that ps cannot obtain their command line parameters (cmline). Generally speaking, in the output of ps, names enclosed in square brackets are usually kernel threads.

Summary #

Interrupt handlers in Linux are divided into the top half and the bottom half:

The top half corresponds to hardware interrupts and is used to quickly handle interrupts.
The bottom half corresponds to soft interrupts and is used to asynchronously handle unfinished work from the top half.

Soft interrupts in Linux include various types such as network transmission and reception, timers, scheduling, RCU locks, etc. The running status of soft interrupts can be observed by viewing /proc/softirqs.

Reflection #

Finally, I would like to discuss with you how you understand soft interrupts. Have you ever encountered performance issues caused by soft interrupts? How did you analyze their bottlenecks? You can summarize your thoughts and write down your own questions, incorporating today’s content.

Feel free to discuss with me in the comments section, and feel free to share this article with your colleagues and friends. Let’s practice in real-life scenarios and improve through communication.