07 Case Study How to Prevent System Crashes Caused by Memory Leaks #
Hello, I’m Shaoyafang.
In the previous lesson, we talked about which types of process memory are more prone to memory leaks. In this lesson, let’s discuss how to deal with the issue of memory leaks.
We know that memory leaks are something that can easily happen, but if they do not pose a threat to the application and system, they will not be harmful. Of course, I’m not saying that you don’t need to be concerned about this kind of memory leaks. For programmers who strive for perfection, it is still necessary to completely resolve them.
However, there are certain memory leaks that require special attention, such as memory leaks in long-running background processes. This kind of leak accumulates over time and gradually consumes system memory, and may even cause system deadlock.
Before we understand the harm caused by memory leaks, let’s take a look at what kind of memory leaks are harmful.
What kind of memory leaks are harmful? #
Here is a simple example program of a memory leak:
#include <stdlib.h>
#include <string.h>
#define SIZE (1024 * 1024 * 1024) /* 1G */
int main()
{
char *p = malloc(SIZE);
if (!p)
return -1;
memset(p, 1, SIZE);
/* Then this block of memory is no longer used */
/* The process exits without freeing the memory pointed to by p */
/* free(p); */
return 0;
}
We can see that in this program, 1G of memory is allocated, but it is not released before the process exits. Is this considered a memory leak?
We can use a simple memory leak checking tool (valgrind) to take a look:
$ valgrind --leak-check=full ./a.out
==20146== HEAP SUMMARY:
==20146== in use at exit: 1,073,741,824 bytes in 1 blocks
==20146== total heap usage: 1 allocs, 0 frees, 1,073,741,824 bytes allocated
==20146==
==20146== 1,073,741,824 bytes in 1 blocks are possibly lost in loss record 1 of 1
==20146== at 0x4C29F73: malloc (vg_replace_malloc.c:309)
==20146== by 0x400543: main (in /home/yafang/test/mmleak/a.out)
==20146==
==20146== LEAK SUMMARY:
==20146== definitely lost: 0 bytes in 0 blocks
==20146== indirectly lost: 0 bytes in 0 blocks
==20146== possibly lost: 1,073,741,824 bytes in 1 blocks
==20146== still reachable: 0 bytes in 0 blocks
==20146== suppressed: 0 bytes in 0 blocks
From the valgrind check result, we can see that the allocated memory was only used once (memset) and was not used again, but it was not freed after use. This is a typical memory leak. Is this memory leak harmful?
To understand the harm caused by this memory leak, let’s talk about the allocation and deallocation of the process address space. Below is a simplified diagram:
From the diagram, we can see that when a process exits, it removes all the mappings it has established. In other words, when a process exits, the memory it has allocated will be freed. Therefore, this memory leak is not harmful. However, it is still recommended to add free(p) in the program to comply with programming conventions. If we modify the program to include free(p) and compile again, and then use valgrind to check, we will find that there is no memory leak:
$ valgrind --leak-check=full ./a.out
==20123== HEAP SUMMARY:
==20123== in use at exit: 0 bytes in 0 blocks
==20123== total heap usage: 1 allocs, 1 frees, 1,073,741,824 bytes allocated
==20123==
==20123== All heap blocks were freed -- no leaks are possible
In summary, if a process does not run for a long time, even if there is a memory leak (such as in this example where only malloc is used without free), it is not harmful because the kernel will free the memory allocated by the process when it exits.
The example we mentioned earlier is a memory leak that is harmless to an application. Let’s continue to see what memory leaks can be harmful to an application. Let’s take malloc as an example and look at a simple example program:
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define SIZE (1024 * 1024 * 1024) /* 1G */
void process_memory()
{
char *p;
p = malloc(SIZE);
if (!p)
return;
memset(p, 1, SIZE);
/* Forget to free this memory */
}
void process_others()
{
sleep(1);
}
int main()
{
/* This memory is only processed once and will not be used again */
process_memory();
/* The process will run for a long time */
while (1) {
process_others();
}
return 0;
}
This is a program that runs for a long time. In process_memory(), we allocate 1G of memory, use it once, and then never use it again. Since this memory will not be used again, it results in memory waste. If there are many such programs, the leaked memory will accumulate, and the available memory in the system will decrease.
For background services, they usually need to run for a long time, so memory leaks in background services can cause actual harm to the system. So, what harm will it cause and how can we deal with it?
How to prevent the harm caused by memory leaks? #
Let’s take the example of the malloc() program mentioned above. In this example, it only allocates 1GB of memory. If memory is continuously allocated without freeing it, you will find that the system memory will quickly be depleted, triggering the OOM killer to kill processes. This information can be viewed by using the dmesg command (which is used to view kernel logs):
$ dmesg
[944835.029319] a.out invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[...]
[944835.052448] Out of memory: Killed process 1426 (a.out) total-vm:8392864kB, anon-rss:7551936kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:14832kB oom_score_adj:0
When the system memory is not sufficient, the OOM killer is awakened to choose a process to kill. In our example, it kills the leaking program. After the process is killed, the entire system becomes safe. However, it’s important to note that the OOM killer selects processes based on a strategy, and it may not necessarily kill the process that is leaking memory, but could potentially kill an innocent process instead. Moreover, OOM itself can also have some side effects.
Let me explain an actual case that occurred in a production environment, which I also reported to the Linux kernel community for improvement. We will discuss it in detail below.
This case is related to the OOM log, which can be understood as a single producer and multiple consumer model, as shown in the following image:
This single producer and multiple consumer model is actually determined by the printk mechanism used by the OOM killer to print logs (OOM info). printk checks where these logs need to be output to, for example, writing them to the kernel buffer, which can then be viewed using the dmesg command. We usually also configure rsyslog, and rsyslogd will dump the contents of the kernel buffer to the log file (/var/log/messages). The server may also be connected to some consoles (console), such as serial ports, where these logs will be outputted.
The problem lies in the console. If the console speed is slow, outputting too many logs can be very time-consuming. At that time, we had configured “console=ttyS1,19200”, which means a low baud rate of 19200 for the serial port. It takes about 10 seconds to print a complete OOM info, which becomes a bottleneck when the system memory is under pressure. Why is it a bottleneck? The answer is shown in the following image:
When process A fails to allocate memory, it triggers OOM, which will print many logs (these logs are for easy analysis of why OOM occurred), and then select an appropriate process to kill, thereby freeing up idle memory that can be used for subsequent memory allocation.
If the OOM process takes a long time (i.e., the time required to print to the slow console is too long, as shown in the red part of the image above), other processes (process B) may also request memory at the same time, and they will also fail to allocate memory. As a result, process B also triggers OOM to attempt to release memory. However, OOM has a global lock (oom_lock) to protect it, and when process B tries to acquire (trylock) this lock, it fails and has to retry.
If there are many processes in the system that are requesting memory at the same time, all these processes will be blocked here, creating a vicious cycle and even causing the system to be unresponsive for a long time (hang). Regarding this issue, I had some discussions with Michal Hocko, the maintainer of the Linux kernel memory subsystem, and Tetsuo Handa, the active developer of the OOM (Out of Memory) sub-module. However, we did not come up with a perfect solution. Currently, there are only some mitigation measures available, as follows:
- Reduce the amount of information printed when OOM occurs - By adjusting
vm.oom_dump_tasksto 0, we can avoid dumping information about all killable processes in the current system. If there are many processes, printing this information can be time-consuming. In our case, this process took about 6 seconds, which accounted for more than half of the total 10 seconds of the OOM operation. Therefore, reducing the printing of this information can alleviate the problem.
However, this is not a perfect solution, but only a mitigation measure. When we set vm.oom_dump_tasks to 1, we can use the printed information to check whether the OOM killer has selected reasonable processes to kill and whether there are any unreasonable OOM configuration strategies in the system. If we set it to 0, we will not have access to this information. Furthermore, these messages will not be printed to the console or the kernel buffer, preventing them from being dumped into a log file that does not cause problems.
-
Adjust the console logging level to prevent OOM messages from being printed to the console - By adjusting
/proc/sys/kernel/printk, we can avoid outputting OOM messages to the console. By settingconsole_loglevelto a value lower than the OOM log level (which is 4), we can prevent OOM messages from being printed to the console. For example, we can set it to 3:# Initial configuration (set to 7): All information will be output to the console $ cat /proc/sys/kernel/printk 7 4 1 7 # Adjust console_loglevel to prevent OOM messages from being printed to the console $ echo "3 4 1 7" > /proc/sys/kernel/printk # Check the adjusted configuration $ cat /proc/sys/kernel/printk 3 4 1
However, doing so will prevent all kernel logs below the default level (which is 4) from being output to the console. In situations where the system encounters problems, sometimes we need to check the console information (such as when we cannot log in to the server) to determine the cause of the problem. If certain information is not printed to the console, it may affect our analysis.
Both of these mitigation measures have their advantages and disadvantages. You need to choose based on your specific situation. If you are unsure how to choose, I suggest you go with the second approach because the probability of using the console is relatively low, so the impact of the second solution is relatively minor.
After the OOM-related logging outputs are dealt with, we move on to the next stage: selecting the most necessary process to kill. The process of selecting the process to kill is also a complex one, and improper configurations can cause other issues. We will analyze this part in the next lesson.
Class Summary #
In this class, we talked about what memory leaks are and the potential harm they can cause. For long-running background tasks, memory leaks can pose a serious threat to the system, so it is important to pay attention to memory leaks in these tasks.
Memory leaks are very common, so we need to be prepared for them in advance: even if there is a leak, we should prevent it from causing significant harm to the system. Most long-term memory leaks will eventually result in an Out of Memory (OOM) error, so you need to understand the relevant knowledge about OOM to handle this backup work effectively.
If your server has slow serial devices, you must prevent them from receiving too many logs, especially logs generated by OOM. The volume of OOM logs is large, and printing the complete OOM information can be time-consuming, causing blocking of processes that are requesting memory, and even causing the entire system to freeze.
Murphy’s Law tells us that if something can go wrong, no matter how small the possibility, it will definitely happen. Applied to memory leaks, it means that once your system becomes complex enough, they will inevitably occur. Therefore, when dealing with memory leaks, you must not only take preventive measures but also be prepared for the potential harm they can cause.
Homework #
Please write some applications to create test cases for memory leaks, and then use valgrind to observe them. Feel free to share your thoughts in the comments section.
Thank you for your reading. If you find this lesson helpful, please share it with your friends. See you in the next lecture.