06 Basics Which Memory Types in Processes Are Prone to Memory Leaks

06 Basics Which Memory Types in Processes are Prone to Memory Leaks #

Hello, I’m Shao Yafang. Today, we will enter the second module of the course to discuss the topic of memory leaks.

I believe that in your daily work, you may have encountered the following scenarios:

  • As the background tasks in the server continue to run, the available memory in the system becomes less and less.
  • The application is suddenly killed by OOM while running.
  • The process doesn’t seem to consume much memory, but the system still runs out of memory.
  • …

Similar problems are likely to be caused by memory leaks. We all know that memory leaks refer to memory that has been allocated but not released, resulting in this portion of memory being unable to be used again. Moreover, even worse, the pointers pointing to this memory space no longer exist, making it impossible to access this memory space anymore.

The memory leaks we typically encounter may be memory leaks in applications or memory leaks in the kernel (operating system). Memory leaks in applications may be leaks in heap memory or leaks in memory mapping regions. These different types of memory leaks have different manifestations and require different solutions. Therefore, to better handle memory leak issues, we need to first understand these different types of memory.

These different types of memory can all be understood as part of the process’s address space. So how does the address space work?

Process Address Space #

We use a diagram to represent the address space of a process. The left side of the diagram shows how a process can change its virtual address space, while the middle shows how the virtual address space is divided. The right side represents the physical memory or physical address space corresponding to the process’s virtual address space.

Let’s talk about this process in more detail.

First, the application program calls functions related to memory allocation and deallocation, such as malloc(3), free(3), calloc(3), or directly uses system calls such as mmap(2), munmap(2), brk(2), sbrk(2).

If a library function is used, these functions actually encapsulate system calls, so it can be understood that the application program dynamically allocates and deallocates memory, and ultimately goes through system calls such as mmap(2), munmap(2), brk(2), sbrk(2). Of course, there are some memory optimizations performed by these libraries, such as malloc(3) may call mmap(2) or brk(2).

Then, these memory-related system calls will modify the process’s address space. brk(2) and sbrk(2) modify the heap, while mmap(2) and munmap(2) modify the Memory Mapping Region.

Please note that these operations are performed on virtual addresses. Application programs deal with virtual addresses and do not directly deal with physical addresses. Ultimately, virtual addresses need to be translated into physical addresses. Since Linux uses pages for management, this process is called Paging.

Let’s use a table to summarize the different memory types corresponding to different allocation methods. This table also includes Page Cache discussed in our previous module, so you can consider it as a comprehensive summary of memory allocation types by a process:

There are many terms involved in this table. We will briefly introduce the important parts, and see which parts in this table may cause memory leaks.

There are many types of memory required for a process to run. In general, these types of memory can be distinguished from whether they are file mappings and whether they are private memory. They can be divided into the four categories listed above.

  • Private Anonymous Memory. The process’s heap, stack, and memory allocated using mmap(MAP_ANON | MAP_PRIVATE) all belong to this type of memory. The stack is managed by the operating system, and the application program does not need to be concerned with its allocation and deallocation. The application program (programmer) is responsible for managing the heap and private anonymous mappings, so they are areas where memory leaks are likely to occur.
  • Shared Anonymous Memory. Memory allocated by the process using mmap(MAP_ANON | MAP_SHARED), such as tmpfs and shm, belongs to this type of memory. This type of memory is also managed by the application program, so memory leaks are also possible.
  • Private File Mapping. Memory allocated by the process using mmap(MAP_FILE | MAP_PRIVATE), such as mapping shared libraries and code segments of executable files into its address space, belongs to this type of memory. For the mapping of shared libraries and code segments of executable files, this is managed by the operating system, and the application program does not need to be concerned with their allocation and deallocation. Memory directly allocated by the application program using mmap(MAP_FILE | MAP_PRIVATE) needs to be managed by the application program itself, and this is where memory leaks may occur.
  • Shared File Mapping. Memory allocated by the process using mmap(MAP_FILE | MAP_SHARED), such as the File Page Cache discussed in the previous module, belongs to this type of memory. This part of the memory also needs to be allocated and deallocated by the application program, so there is a possibility of memory leaks.

After understanding these different types of memory in the process’s virtual address space, let’s take a look at the corresponding physical memory.

As mentioned earlier, the process’s virtual address space is mapped to physical memory through Paging. Memory allocated by the process through malloc() or mmap() is virtual memory, and physical memory is only allocated when data (e.g., using memset) is actually written to these addresses.

You may wonder if the process calls malloc() or mmap() without writing to these addresses, thus not allocating physical memory, do we need to worry about memory leaks?

The answer is that memory leaks still need to be considered because this can lead to the exhaustion of the process’s virtual address space, i.e., virtual address space also has memory leak problems. We will analyze corresponding cases in the next section, so we won’t go into details here.

Next, let’s take a closer look at the Paging process using an image.

As shown in the above image, the Paging process is roughly as follows: the CPU passes the virtual address it wants to access to the MMU (Memory Management Unit), and the MMU first looks up the translation relationship in the Translation Lookaside Buffer (TLB), which is a cache for page tables. If the corresponding physical address is found, it is accessed directly. If not, it is searched and computed in the page table. Finally, the virtual address accessed by the process corresponds to the actual physical address.

After understanding the related knowledge of address space, you can make a reasonable plan or control the address space of a process. In this way, when problems occur, they will not have too severe consequences. You can understand the planned address space of a process as a fallback solution for memory problems. The most typical way to plan the address space of a process on Linux is through ulimit. You can adjust it to plan the maximum virtual address space, physical address space, stack space, and so on for a process.

This is all for the knowledge related to process address space. Next, let’s see how to use tools to observe the address space of a process.

Observing Process Memory with Data #

Learning to observe the process address space is a prerequisite for analyzing memory leak problems. When you suspect a memory leak, you need to first observe which memory is continuously growing and which memory is unusually large. This way, you can determine roughly where the memory leak occurs and analyze it accordingly. On the contrary, if you blindly guess where the problem lies without carefully observing the process address space, dealing with the problem may waste a lot of time, or even lead you in the wrong direction.

So what tools can we use to observe the process? The commonly used tools for observing process memory, such as pmap, ps, and top, can all be used to observe process memory effectively.

Firstly, we can use top to observe the overall memory usage of all processes in the system. After opening top, press “g” and then enter “3” to enter memory mode. In memory mode, we can see the %MEM, VIRT, RES, CODE, DATA, SHR, nMaj, and nDRT of each process. These pieces of information are obtained by tracing the top process using strace. You will find that these pieces of information are read from the /proc/[pid]/statm and /proc/[pid]/stat files:

$ strace -p `pidof top`
open("/proc/16348/statm", O_RDONLY)     = 9
read(9, "40509 1143 956 24 0 324 0\n", 1024) = 26
close(9)                                = 0
...
open("/proc/16366/stat", O_RDONLY)      = 9
read(9, "16366 (kworker/u16:1-events_unbo"..., 1024) = 182
close(9)
...

Except for nMaj (Major Page Fault, which refers to the number of pages that are not in memory and need to be read from disk), %MEM is derived from RES, and the other memory information is read from the statm file. The following is the correspondence between fields in the top command and fields in the statm file:

Furthermore, if you observe carefully, you may find that sometimes the sum of RES of all processes will be larger than the total physical memory of the system. This is because some memory in RES is shared by certain processes.

After understanding the memory usage of each process in the system, if you want to further examine the memory usage details of a specific process, you can use pmap. The following is an example of pmap displaying parts of the address space of the sshd process:

$  pmap -x `pidof sshd`
Address           Kbytes     RSS   Dirty Mode  Mapping 
000055e798e1d000     768     652       0 r-x-- sshd
000055e7990dc000      16      16      16 r---- sshd
000055e7990e0000       4       4       4 rw--- sshd
000055e7990e1000      40      40      40 rw---   [ anon ]
...
00007f189613a000    1800    1624       0 r-x-- libc-2.17.so
00007f18962fc000    2048       0       0 ----- libc-2.17.so
00007f18964fc000      16      16      16 r---- libc-2.17.so
00007f1896500000       8       8       8 rw--- libc-2.17.so
...
00007ffd9d30f000     132      40      40 rw---   [ stack ]
...

Each line represents a type of memory (Virtual Memory Area, VMA), and each column means the following.

  • Mapping: indicates the file that occupies memory in the file mapping, such as the executable file sshd, or the heap, or the stack, or others.
  • Mode: indicates the permission of this memory. For example, “r-x” means readable and executable, which usually refers to the code segment (text segment); “rw-” means readable and writable, which usually refers to the data segment; “r–” means read-only, which usually denotes the read-only part of the data segment.
  • Address, Kbytes, RSS, Dirty: Address and Kbytes respectively indicate the starting address and the size of virtual memory. RSS (Resident Set Size) indicates the size of physical memory allocated in virtual memory. Dirty indicates the number of bytes in memory that are not synchronized to disk.

As you can see, pmap allows us to clearly observe the entire address space of a process, including the amount of physical memory allocated. This is very helpful for making a rough judgment on the memory usage of a process. For example, if the [heap] in the address space is too large, it may indicate a heap memory leak. Additionally, if the process address space contains too many VMAs (you can consider each line in maps as a VMA), it is likely that the application calls a lot of mmap without munmap. Furthermore, by continuously observing the changes in the address space, if you find certain items consistently growing, it is likely that there is a problem there.

pmap also parses files in /proc, specifically the /proc/[pid]/maps and /proc/[pid]/smaps files, where the smaps file provides more detailed information compared to the maps file. You can compare the output of /proc/[pid]/maps and pmap, and you will find that their contents are consistent.

In addition to observing the memory of the process itself, we can also observe the relationship between the memory allocated by the process and system metrics. Let’s take the commonly used /proc/meminfo as an example to illustrate how the four types of memory (private anonymous, private file, shared anonymous, shared file) we mentioned earlier are reflected in system metrics.

As shown in the above figure, any private memory is reflected in the AnonPages field in /proc/meminfo, and any shared memory is reflected in the Cached field. Anonymous shared memory is also reflected in the Shmem field.

$ cat /proc/meminfo
...
Cached:          3799380 kB
...
AnonPages:       1060684 kB
...
Shmem:              8724 kB
...

Similarly, I recommend you to write some test cases to observe it yourself. This way, your understanding will be deeper.

This is all for the basic knowledge of memory management related to processes. In the next lesson, we will discuss actual cases of memory leaks and their dangers.

Class Summary #

In this class, we discussed some knowledge related to process memory management, including the virtual memory and physical memory of a process. The key points are as follows.

  • The addresses that processes directly read and write are virtual addresses, and these virtual addresses are eventually converted into physical memory addresses through the process of Paging, which is completed by the kernel.
  • The memory types of a process can be classified into four different types, which are anon (anonymous) and file (file), private and shared. All memory related to a process is a combination of these four types.
  • When checking the memory of a process, you can first use “top” to view the overall memory usage of various processes in the system, and then use “pmap” to observe the memory details of a specific process.

Process memory management involves many terms. For some commonly used terms, such as VIRT, RES, SHR, etc., you still need to remember their meanings. Only when you are familiar with their meanings, you will be more proficient in analyzing memory issues. For example, if RES is high while SHR is not, it may indicate a heap memory leak. If SHR is high, it may indicate the continuous growth of data such as tmpfs/shm. If VIRT is high while RES is small, it may indicate that the process is continuously requesting memory but not performing any read or write operations on these memory regions, which indicates a memory leak in the virtual address space.

Similarly, I hope you can write some test cases yourself to observe the changes in these indicators.

Homework #

After class, you can write some test programs to allocate the four different types of memory mentioned in this class, observe the changes in the process address space, and the changes in system memory metrics. Feel free to discuss with me in the comments.

Thank you for reading. If you found this class helpful, please feel free to share it with your friends. See you in the next lesson.