20 Why Is the Memory Utilization Still High After Data Deletion

20 Why is the Memory Utilization Still High After Data Deletion #

When using Redis, we often encounter a problem: even though we have deleted data and the data size is no longer large, why does the top command show that Redis still occupies a lot of memory?

In fact, this is because when data is deleted, the memory space released by Redis is managed by the memory allocator and is not immediately returned to the operating system. Therefore, the operating system still records that a large amount of memory has been allocated to Redis.

However, this often comes with a potential risk: the released memory space of Redis may not be contiguous. As a result, these non-contiguous memory spaces are likely to be in an idle state. This leads to a problem: although there is free space, Redis cannot use it to store data. This not only reduces the amount of data that Redis can actually store, but also reduces the cost-effectiveness of running Redis.

Let me use an analogy. We can compare the memory space of Redis to the seating capacity of a high-speed train carriage. If a high-speed train has a large number of seats but few passengers are being transported, then the efficiency of running the high-speed train is low, the cost is high, and the cost-effectiveness is reduced. The same goes for Redis. If you happen to rent a 16GB cloud server to run Redis, but only save 8GB of data, then the cost-effectiveness of renting this cloud server will be halved, which is definitely not what you want.

Therefore, in this lesson, I will discuss the issue of storage efficiency of Redis memory space and explore why the memory is idle even after data deletion, as well as corresponding solutions.

What is memory fragmentation? #

In many cases, when there is idle memory space, it is often due to a more serious memory fragmentation in the operating system. So, what is memory fragmentation?

To help you understand, I will use the example of seat arrangements on a high-speed train. Let’s say there are a total of 60 seats in a train car, and 57 tickets have already been sold. You and two friends want to take the high-speed train for a trip, and you need three tickets. However, you want to sit together so that you can chat during the journey. But when selecting seats, you find that there are no consecutive seats available. So, you have to change to another train. As a result, you need to change your travel time, and three seats in this train car are left vacant.

In fact, the vacant seats in this train are matching the number of people in your group, but these vacant seats are scattered as shown in the following figure:

We can call these scattered vacant seats “seat fragments”. Once you understand this point, it is easy to understand memory fragmentation in the operating system. Although the total amount of remaining memory space in the operating system is sufficient, when an application requests for a continuous address space of N bytes, there is no continuous space of size N bytes available in the remaining memory space. Thus, these remaining spaces are memory fragments (such as the “2 bytes free” and “1 byte free” shown in the above figure).

So, what causes memory fragmentation in Redis? Let’s take a closer look next. Only by understanding the causes of memory fragmentation can we take appropriate measures to fully utilize the memory space occupied by Redis and increase the amount of data stored.

How are memory fragments formed? #

In fact, the formation of memory fragments has two underlying factors: internal factors and external factors. Simply put, the internal factor refers to the memory allocation mechanism of the operating system, while the external factor refers to the load characteristics of Redis.

Internal factor: Allocation strategy of memory allocator #

The allocation strategy of the memory allocator determines that the operating system cannot achieve “on-demand allocation.” This is because the memory allocator generally allocates memory in fixed sizes, rather than allocating memory to programs based on the actual size requested by the application.

Redis can use multiple memory allocators such as libc, jemalloc, and tcmalloc to allocate memory, with jemalloc being the default. Next, I will explain specifically using jemalloc as an example, although other allocators have similar issues.

One of the allocation strategies of jemalloc is to divide memory space into a series of fixed sizes, such as 8 bytes, 16 bytes, 32 bytes, 48 bytes, …, 2KB, 4KB, 8KB, and so on. When a program requests memory that is closest to a certain fixed value, jemalloc will allocate a space of the corresponding size.

This allocation method itself aims to reduce the number of allocations. For example, if Redis requests a 20-byte space to store data, jemalloc will allocate 32 bytes. If the application needs to write another 10 bytes of data, Redis does not need to allocate additional space from the operating system because the previously allocated 32 bytes is sufficient, thereby avoiding an allocation operation.

However, if Redis requests memory of different sizes each time, this allocation method may result in fragmentation, which is precisely caused by external factors of Redis.

External factors: Variations in key-value sizes and update/delete operations #

Redis is often used as a shared caching system or key-value database to provide services to different business applications. Therefore, data of different sizes from different applications may be stored in Redis. As a result, when Redis requests memory allocation, there are inherently varying space requirements. This is the first external factor.

However, as we mentioned earlier, memory allocators can only allocate memory in fixed sizes. Therefore, the allocated memory space generally differs from the requested space, leading to a certain degree of fragmentation and reduced storage efficiency.

For example, if application A stores 6 bytes of data and jemalloc allocates 8 bytes according to its allocation strategy, the extra 2 bytes of space becomes memory fragments, as shown in the figure below:

The second external factor is that these key-value pairs are updated and deleted, resulting in space expansion and deallocation. Specifically, on one hand, if the modified key-value pair becomes larger or smaller, additional space needs to be occupied or unused space needs to be released. On the other hand, when a key-value pair is deleted, the corresponding memory space is no longer needed, and it creates free space.

To help you understand, I’ve created the following diagram:

Initially, applications A, B, C, and D respectively store data of 3, 1, 2, and 4 bytes and occupy the corresponding memory spaces. Then, application D deletes 1 byte, freeing up that 1-byte memory space. Subsequently, application A modifies its data, increasing it from 3 bytes to 4 bytes. In order to maintain the spatial continuity of A’s data, the operating system needs to copy B’s data to another space, such as the space just freed up by D. Meanwhile, applications C and D also delete data of 2 and 1 bytes, respectively, resulting in 2-byte and 1-byte free fragments in the entire memory space. If application E wants a continuous space of 3 bytes, it cannot be satisfied because although there is enough space in total, it is fragmented space and not continuous.

Alright, up to this point, we have learned about the internal and external factors that cause memory fragmentation. Among them, the memory allocator strategy is an internal factor, while the load of Redis belongs to an external factor, including key-value pairs of varying sizes and memory space changes caused by updating or deleting key-value pairs.

The existence of a large number of memory fragments can reduce the actual utilization rate of Redis memory. Next, we will address this issue. However, before solving the problem, we need to determine whether there are memory fragments during the running process of Redis.

How to Determine if there is Memory Fragmentation? #

Redis is an in-memory database, and the efficiency of Redis directly relates to the utilization of memory. In order to allow users to monitor real-time memory usage, Redis provides the INFO command to query detailed memory usage information. The command is as follows:

INFO memory
# Memory
used_memory: 1073741736
used_memory_human: 1024.00M
used_memory_rss: 1997159792
used_memory_rss_human: 1.86G
…
mem_fragmentation_ratio: 1.86

Here, there is a metric called mem_fragmentation_ratio, which represents the current memory fragmentation ratio of Redis. So how is this fragmentation ratio calculated? In fact, it is the result of dividing the two metrics used_memory_rss and used_memory from the command above.

mem_fragmentation_ratio = used_memory_rss / used_memory

used_memory_rss is the physical memory allocated by the operating system to Redis, including fragmentation, while used_memory is the actual space used by Redis to store data.

Let me give you a simple example. For example, Redis requests to use 100 bytes (used_memory), but the operating system actually allocates 128 bytes (used_memory_rss). In this case, the mem_fragmentation_ratio is 1.28.

Now that we know this metric, how should we use it? Here, I provide some experience thresholds:

mem_fragmentation_ratio greater than 1 but less than 1.5. This situation is reasonable. This is because the factors I introduced earlier are difficult to avoid. After all, the internal memory allocator must be used and the allocation strategy is universal and not easily modified. The external factors are determined by the Redis load and cannot be restricted. Therefore, the existence of memory fragmentation is normal.
mem_fragmentation_ratio greater than 1.5. This indicates that the memory fragmentation ratio has exceeded 50%. Under normal circumstances, at this point, we need to take some measures to reduce the memory fragmentation ratio.

How to clean up memory fragmentation? #

When Redis experiences memory fragmentation, a “wild and crude” method is to restart the Redis instance. However, this is not an “elegant” solution because restarting Redis has two consequences:

If the data in Redis is not persisted, the data will be lost.
Even if the Redis data is persisted, we still need to recover through AOF or RDB, and the recovery time depends on the size of AOF or RDB. If there is only one Redis instance, the recovery phase cannot provide service.

So, is there any other good solution?

Fortunately, starting from version 4.0-RC3, Redis itself provides a method for automatically cleaning up memory fragmentation. Let’s start by looking at the basic mechanism of this method.

Memory fragmentation cleaning, to put it simply, is “moving to merge space.”

I’ll use the example of selecting seats in a high-speed train car to explain. You and your friends don’t want to waste time, so you bought three tickets with seats that are not together. However, after boarding, you and your friends will switch seats with others and sit together again.

With this analogy, the mechanism of memory fragmentation cleaning is easy to understand. When data divides a continuous memory space into several non-continuous spaces, the operating system will copy the data elsewhere. At this time, the data copy needs to empty the space previously occupied by these data, and turn the originally non-continuous memory space into a continuous space. Otherwise, if, after the data copy, a continuous memory space is not formed, this cannot be considered as cleaning.

Let me draw a diagram to explain.

Before performing fragmentation cleaning, there are 1 byte of 2-byte free space and 1 byte of 1-byte free space in this 10-byte space, but these two spaces are not continuous. When the operating system cleans up the fragmentation, it first copies the data of application D to the 2-byte free space and releases the space originally occupied by D. Then, it copies the data of B to the original space of D. In this way, the last three bytes of this 10-byte space become a continuous space. At this point, the fragmentation cleaning is complete.

However, it should be noted that fragmentation cleaning comes at a cost. The operating system needs to copy multiple copies of data to new locations and free up the original space, which incurs time overhead. Because Redis is single-threaded, when data is copied, Redis has to wait, which means Redis cannot process requests in a timely manner, resulting in reduced performance. Furthermore, sometimes, data copying needs to be done in a specific order, just like the example of cleaning up memory fragmentation, the operating system needs to copy D and release D’s space before copying B. This requirement for order further increases Redis’s waiting time, leading to reduced performance.

So, is there any way to alleviate this problem as much as possible? This leads to the mention of the parameters specifically set by Redis for the automatic memory fragmentation cleaning mechanism. We can control the start and end time of fragmentation cleaning, as well as the CPU usage, which reduces the performance impact of fragmentation cleaning on Redis’s request handling.

First, Redis needs to enable automatic memory fragmentation cleaning, which can be done by setting the “activedefrag” configuration item to “yes”, as follows:

config set activedefrag yes

This command only enables the automatic cleaning function, but the specific time to clean up will be controlled by the following two parameters. These two parameters set a condition for triggering memory cleaning. If these two conditions are met at the same time, the cleaning starts. During the cleaning process, if one condition is not met, the automatic cleaning stops.

active-defrag-ignore-bytes 100mb: It means that when the bytes of memory fragmentation reach 100MB, the cleaning starts.
active-defrag-threshold-lower 10: It means that when the proportion of memory fragmentation space to the total space allocated to Redis by the operating system reaches 10%, the cleaning starts.

To minimize the performance impact of fragmentation cleaning on Redis’s normal request handling, the automatic memory fragmentation cleaning mechanism also monitors the CPU time consumed by the cleaning operation. Two parameters are set to control the upper and lower limits of the CPU time consumed by the cleaning operation, which ensures that the cleaning work can proceed normally and avoids blocking Redis with a large amount of memory copying, resulting in increased response latency. These two parameters are as follows:

active-defrag-cycle-min 25: It means that the proportion of CPU time used by the automatic cleaning process should not be lower than 25%, ensuring that the cleaning can proceed normally.
active-defrag-cycle-max 75: It means that the proportion of CPU time used by the automatic cleaning process should not exceed 75%. Once it exceeds, the cleaning stops to avoid blocking Redis and causing a high response delay during cleaning.

The automatic memory fragmentation cleaning mechanism considers not only the proportion of fragmentation, the impact on the efficiency of Redis memory usage, but also the proportion of CPU time consumed by the cleaning mechanism and the performance impact on Redis. Moreover, the cleaning mechanism also provides four parameters, allowing us to flexibly use them according to the actual data volume requirements and performance requirements in practice. It is recommended that you make good use of this mechanism in practice.

Summary #

In this lesson, we have learned about the memory efficiency issues of Redis, with a key focus on recognizing and dealing with memory fragmentation. In simple terms, there are “three ones”:

The info memory command is a good tool that helps you view the fragmentation rate.
The fragmentation rate threshold is a good guideline that helps you determine whether a fragmentation cleanup is necessary.
Automatic memory fragmentation cleanup is a good approach that avoids reducing the actual memory utilization of Redis due to fragmentation, improving cost-effectiveness.

Memory fragmentation is not something to be feared. What we need to do is understand it, give it importance, and use efficient methods to solve it.

Lastly, I would like to provide you with a tip: Automatic memory fragmentation cleanup involves memory copying, which poses a potential risk for Redis. If you encounter a slowdown in Redis performance during implementation, remember to check the logs to see if a fragmentation cleanup is in progress. If Redis is indeed performing fragmentation cleanup, I recommend reducing the value of active-defrag-cycle-max to mitigate the impact on normal request processing.

One question per lesson #

As usual, I have a small question for you. In this lesson, I mentioned that we can use mem_fragmentation_ratio to determine whether Redis’s current memory fragmentation ratio is serious, and the threshold I provided is greater than 1. Now, I would like you to discuss what happens if mem_fragmentation_ratio is less than 1. What impact does it have on Redis’s performance and memory utilization?

Feel free to write down your thoughts and answer in the comment section, and let’s have a discussion. If you find today’s content helpful, please feel free to share it with your friends or colleagues. See you in the next lesson.