18 Case Study Memory Leak, How to Locate and Handle

18 Case Study Memory Leak, How to Locate and Handle #

Hello, I am Ni Pengfei.

Through the previous sections on memory basics, I believe you have gained a preliminary understanding of how Linux memory works.

For a normal process, what can be seen is actually the virtual memory provided by the kernel. This virtual memory needs to be mapped to physical memory by the system through the page table.

When a process requests virtual memory through malloc(), the system does not immediately allocate physical memory for it. Instead, memory allocation is done in the kernel through a page fault exception when the memory is first accessed.

To coordinate the performance difference between the CPU and the disk, Linux also uses Cache and Buffer to cache data for files and disk reads/writes in memory.

For an application, dynamic memory allocation and deallocation are both core and complex logical functions. In the process of managing memory, various “accidents” can easily occur, such as:

  • Failure to properly deallocate allocated memory, resulting in leaks.

  • Accessing addresses outside the allocated memory boundaries, causing program exceptions, and so on.

Today, I’ll show you how memory leaks occur and how to troubleshoot and locate them.

When it comes to memory leaks, we need to start with memory allocation and deallocation.

Memory Allocation and Deallocation #

Let’s start by reviewing which methods are used to allocate memory in an application and how to release it back to the system when done.

As mentioned earlier when discussing memory space in processes, user space memory consists of multiple memory segments, such as the read-only segment, data segment, heap, stack, and file mapping segment. These memory segments are the basic way in which applications use memory.

For example, if you define a local variable in your program, such as an integer array int data[64], you are defining a memory segment that can store 64 integers. Since this is a local variable, it will be allocated from the stack space in memory.

Stack memory is automatically allocated and managed by the system. Once the program goes out of the scope of this local variable, the stack memory will be automatically reclaimed by the system, thus avoiding memory leaks.

In many cases, we may not know the size of the data in advance, so we need to dynamically allocate memory using the standard library function malloc() in our program. In such cases, the system will allocate memory from the heap space in memory.

Heap memory is allocated and managed by the application itself. Unless the program exits, this heap memory will not be automatically released by the system. Instead, the application needs to explicitly call the library function free() to release it. If the application fails to correctly release heap memory, it will result in memory leaks.

These are examples of the stack and heap memory allocation. What about other memory segments? Based on our previous learning, this question isn’t difficult to answer.

  • The read-only segment, including the program’s code and constants, does not allocate new memory since it is read-only, so it does not cause memory leaks.

  • The data segment, including global variables and static variables, has a predetermined size at the time of definition, so it also does not cause memory leaks.

  • The last memory mapping segment, including dynamic link libraries and shared memory, involves dynamic allocation and management by the program. If the program forgets to deallocate after allocation, it will lead to a leak similar to heap memory.

Memory leaks are highly harmful. Memory that is not released can neither be accessed by the application itself nor be reallocated to other applications by the system. Accumulated memory leaks can even exhaust system memory.

Although the system can eventually kill a process using the Out of Memory (OOM) mechanism, the process may have caused a series of reactions prior to OOM, resulting in severe performance issues.

For instance, other processes in need of memory may fail to allocate new memory. Memory shortage can trigger the system’s cache evictions and SWAP mechanisms, further resulting in performance issues related to I/O, and so on.

Given the significant harm caused by memory leaks, how can we detect these issues? In particular, if you have already discovered a memory leak, how can you locate and handle it?

Next, we will use an example of calculating the Fibonacci sequence to explore the methods for locating and resolving memory leak issues.

The Fibonacci sequence is a sequence of numbers: 0, 1, 1, 2, 3, 5, 8, … where the first two numbers are 0 and 1, and each subsequent number is the sum of the previous two numbers. Mathematically, it can be represented as F(n) = F(n-1) + F(n-2) (n >= 2), with F(0) = 0 and F(1) = 1.

Case Study #

Today’s case study is based on Ubuntu 18.04, but it also applies to other Linux systems.

  • Machine configuration: 2 CPUs, 8GB memory

  • Pre-install sysstat, Docker, and bcc packages, for example:

    install sysstat docker #

    sudo apt-get install -y sysstat docker.io

    Install bcc #

    sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 4052245BD4284CDD echo “deb https://repo.iovisor.org/apt/bionic bionic main” | sudo tee /etc/apt/sources.list.d/iovisor.list sudo apt-get update sudo apt-get install -y bcc-tools libbcc-examples linux-headers-$(uname -r)

We are already familiar with sysstat and Docker. The vmstat in the sysstat package can be used to observe the changes in memory; Docker can run the example program.

We have also introduced the bcc package before, which provides a collection of Linux performance analysis tools commonly used to dynamically trace the behavior of processes and the kernel. You don’t need to delve into more details about how it works for now; we will gradually learn about it later. All you need to remember is that after installing it according to the steps above, all the tools it provides will be located in the directory /usr/share/bcc/tools.

Note: bcc-tools requires a kernel version of 4.1 or higher. If you are using CentOS 7 or other systems with a relatively old kernel version, you need to manually upgrade the kernel before installing.

Open a terminal and SSH into the machine to install the above tools.

As with previous cases, all the commands below are assumed to be run with root user privilege. If you are logged into the system as a regular user, run the command sudo su root to switch to the root user.

If you encounter any issues during the installation process, I encourage you to search for solutions yourself. If you can’t find a solution, feel free to ask me in the comments. If you have already installed them before, you can ignore this.

After the installation is complete, run the following command to run the case:

$ docker run --name=app -itd feisky/app:mem-leak

After the case is successfully executed, you need to enter the following command to confirm that the case application has been started correctly. If everything is normal, you should see the following output:

$ docker logs app
2th => 1
3th => 2
4th => 3
5th => 5
6th => 8
7th => 13

From the output, we can see that this case will output a series of values from the Fibonacci sequence. In fact, these values are output every 1 second.

Knowing this, how can we check the memory situation to determine if there is a memory leak? The first tool you might think of is top, but although top can observe the memory usage of the system and processes, it is not suitable for today’s case. When it comes to memory leaks, we should pay more attention to the changing trend of memory usage.

So, as mentioned earlier, today I recommend another familiar tool, vmstat.

Run the following vmstat, wait for a while, and observe the changes in memory. If you forget the meaning of each metric in vmstat, review the previous content or execute man vmstat to query.

# Output a set of data every 3 seconds
$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0      0 6601824  97620 1098784    0    0     0     0   62  322  0  0 100  0  0
0  0      0 6601700  97620 1098788    0    0     0     0   57  251  0  0 100  0  0
0  0      0 6601320  97620 1098788    0    0     0     3   52  306  0  0 100  0  0
0  0      0 6601452  97628 1098788    0    0     0    27   63  326  0  0 100  0  0
2  0      0 6601328  97628 1098788    0    0     0    44   52  299  0  0 100  0  0
0  0      0 6601080  97628 1098792    0    0     0     0   56  285  0  0 100  0  0

From the output, you can see that the free column of memory is constantly changing and is decreasing overall, while the buffer and cache values remain relatively unchanged.

The amount of free memory is gradually decreasing while the buffer and cache remain mostly unchanged. This indicates that the used memory in the system is continuously increasing. However, this does not necessarily indicate a memory leak, as the memory needed by the running application may also increase. For example, if the program uses a dynamically growing array to cache calculation results, the memory usage will naturally increase.

So how can we determine if there is a memory leak? In other words, is there a simple way to find the process that is causing the memory to increase and identify where the growing memory is being used?

Based on the previous content, you should think of using top or ps to observe the memory usage of the processes, and then find the processes with continuously increasing memory usage, and finally use pmap to view the memory distribution of the processes.

However, this method is not very convenient because you would need to write a script to process the output of top or ps in order to judge the changes in memory.

Here, I will introduce a tool specifically designed to detect memory leaks, called memleak. memleak can trace memory allocations and deallocations in the system or specific processes, and periodically output a summary of the unreleased memory and their corresponding call stacks (default 5 seconds).

Of course, memleak is a tool in the bcc package, which we have installed from the beginning. You can run it by executing /usr/share/bcc/tools/memleak. For example, we can run the following command:

# -a displays the size and address of each memory allocation request
# -p specifies the PID of the application in question
$ /usr/share/bcc/tools/memleak -a -p $(pidof app)
WARNING: Couldn't find .text section in /app
WARNING: BCC can't handle sym look ups for /app
    addr = 7f8f704732b0 size = 8192
    addr = 7f8f704772d0 size = 8192
    addr = 7f8f704712a0 size = 8192
    addr = 7f8f704752c0 size = 8192
    32768 bytes in 4 allocations from stack
        [unknown] [app]
        [unknown] [app]

Please note that the tool memleak is trying to analyze an application called app, but it seems that there are some warnings related to the symbols of this application. start_thread+0xdb [libpthread-2.27.so]

From the output of memleak, we can see that the test application keeps allocating memory, and these allocated addresses are not being freed.

There is an issue here, “Couldn’t find .text section in /app,” so the call stack cannot be output correctly, and the last part of the call stack can only be seen as [unknown].

Why does this error occur? In fact, this is because the test application is running in a container. The memleak tool runs outside the container and cannot directly access the process path /app.

For example, if you run the ls command directly in the terminal, you will find that this path does not exist:

$ ls /app
ls: cannot access '/app': No such file or directory

Similar issues have been mentioned in several solution ideas in the perf usage methods in the CPU module. The simplest solution is to build the same path file and dependent libraries outside the container. This test case only has one binary file, so the problem can be fixed by putting the binary file of the test application into the /app path.

For example, you can run the following commands to copy the app binary file from the container and then run the memleak tool again:

$ docker cp app:/app /app
$ /usr/share/bcc/tools/memleak -p $(pidof app) -a
Attaching to pid 12512, Ctrl+C to quit.
[03:00:41] Top 10 stacks with outstanding allocations:
    addr = 7f8f70863220 size = 8192
    addr = 7f8f70861210 size = 8192
    addr = 7f8f7085b1e0 size = 8192
    addr = 7f8f7085f200 size = 8192
    addr = 7f8f7085d1f0 size = 8192
    40960 bytes in 5 allocations from stack
        fibonacci+0x1f [app]
        child+0x4f [app]
        start_thread+0xdb [libpthread-2.27.so]

This time, we finally see the call stack of memory allocation. It turns out that the memory allocated by the fibonacci() function was not freed.

After locating the source of the memory leak, the next step is to view the source code and find a way to fix it. Let’s take a look at the source code of the test application app.c:

$ docker exec app cat /app.c
...
long long *fibonacci(long long *n0, long long *n1)
{
    // Allocate 1024 spaces for long integers to observe the change in memory
    long long *v = (long long *) calloc(1024, sizeof(long long));
    *v = *n0 + *n1;
    return v;
}


void *child(void *arg)
{
    long long n0 = 0;
    long long n1 = 1;
    long long *v = NULL;
    for (int n = 2; n > 0; n++) {
        v = fibonacci(&n0, &n1);
        n0 = n1;
        n1 = *v;
        printf("%dth => %lld\n", n, *v);
        sleep(1);
    }
}
...

You will find that the child() function calls the fibonacci() function, but does not free the memory returned by fibonacci(). So, to fix the leak problem, you just need to add a release function in the child() function, for example:

void *child(void *arg)
{
    ...
    for (int n = 2; n > 0; n++) {
        v = fibonacci(&n0, &n1);
        n0 = n1;
        n1 = *v;
        printf("%dth => %lld\n", n, *v);
        free(v);    // Free the memory
        sleep(1);
    }
}

I have put the fixed code in app-fix.c and packaged it into a Docker image. You can run the following command to verify if the memory leak has been fixed:

# Clean up the original test application
$ docker rm -f app

# Run the fixed application
$ docker run --name=app -itd feisky/app:mem-leak-fix

# Rerun the memleak tool to check for memory leaks
$ /usr/share/bcc/tools/memleak -a -p $(pidof app)
Attaching to pid 18808, Ctrl+C to quit.
[10:23:18] Top 10 stacks with outstanding allocations:
[10:23:23] Top 10 stacks with outstanding allocations:

Now, we can see that the test application no longer has any remaining memory, proving that our fix work has been successful.

Conclusion #

Let’s summarize the content today.

The user memory space that the application can access consists of read-only sections, data sections, the heap, the stack, and file mapping sections, among others. Among them, the heap memory and file mapping sections need the application to dynamically manage the memory segments, so we must be careful in handling them. We not only need to know how to use the standard library function malloc() to dynamically allocate memory, but also remember to call the library function free() to release them after using the memory.

The case we discussed today is relatively simple, and fixing the memory leak only requires adding a free() call. However, real-world applications are much more complex. For example,

  • malloc() and free() do not always appear together. You need to release memory on every exception handling path and successful path.

  • In a multi-threaded program, memory allocated in one thread may be accessed and freed in another thread.

  • More complexly, implicitly allocated memory in third-party library functions may need to be explicitly released by the application.

Therefore, to avoid memory leaks, the most important thing is to cultivate good programming habits. For example, after allocating memory, always write the memory release code first before developing other logic. As the saying goes, “When you borrow, always return,” only then can things run efficiently and borrowing won’t be difficult anymore.

Of course, if you have completed the development tasks, you can use the memleak tool to check if there are memory leaks in the application during runtime. If you find a memory leak, you can locate the memory allocation position based on the application call stack output by memleak, and then release the memory that is no longer accessed.

Reflection #

Lastly, I have a question for you to ponder.

In today’s case, we fixed the memory leak issue by adding a free() call to release the memory allocated by the fibonacci() function. However, besides this approach, are there any other better ways to fix the issue? Based on what you have learned and your own work experience, I believe you can come up with more and better solutions.

Feel free to leave a comment and discuss with me. Share your answers and takeaways. You are also welcome to share this article with your colleagues and friends. Let’s practice and improve together through practical exercises and discussions.