19 How to Respond to the Wave of Latency Slowdown in Redis After

19 How to Respond to the Wave of Latency Slowdown in Redis - After #

In the previous lesson, I introduced two methods for identifying slow Redis instances: response latency and baseline performance. In addition, I shared two solutions for troubleshooting and resolving issues at the Redis command operation level.

However, what if, during the troubleshooting process, you find that Redis is not executing a large number of slow query commands or concurrently deleting a large number of expired keys? Does that mean we are at a loss?

Of course not! I still have many “tricks up my sleeve” that I’m ready to share with you in this lesson!

If the methods from the previous lesson didn’t work, it means you need to focus on other mechanisms that can affect performance - namely, the file system and the operating system.

Redis persists data to disk, and this process relies on the file system. Therefore, the mechanism by which the file system writes data back to disk directly affects the efficiency of Redis persistence. Additionally, during the persistence process, Redis is still receiving other requests, and the efficiency of persistence also affects the performance of Redis in handling requests.

On the other hand, Redis is an in-memory database, and operations on memory are very frequent. Therefore, the memory mechanism of the operating system directly affects the efficiency of Redis. For example, if Redis runs out of memory, the operating system will initiate a swap mechanism, which will significantly slow down Redis.

Next, I will continue to introduce how to further resolve the issue of slow Redis from these two perspectives.

File System: AOF Mode #

You may wonder why the performance of Redis is related to the file system if it is an in-memory database.

As I mentioned earlier, to ensure data reliability, Redis adopts either the AOF log or RDB snapshot. Among them, the AOF log provides three log writing strategies: no, everysec, always. These three strategies rely on two system calls of the file system, which are write and fsync.

The write system call only needs to write the log record to the kernel buffer and can return, without waiting for the log to be actually written to the disk. On the other hand, fsync needs to return only after the log record is written to the disk, which takes a longer time. The following table shows the system calls performed by the three writing strategies.

When the writing strategy is configured as everysec or always, Redis needs to call fsync to write the log to the disk. However, the specific execution of these two writing strategies is different.

When using everysec, Redis allows the loss of one second of operation records, so the Redis main thread does not need to ensure that every operation record is written to the disk. Moreover, fsync takes a long time to execute, and if it is executed in the Redis main thread, it is prone to block the main thread. Therefore, when the writing strategy is configured as everysec, Redis uses a background sub-thread to asynchronously complete the fsync operation.

For the always strategy, Redis needs to ensure that every operation record is written to the disk. If this is done asynchronously by a background sub-thread, the main thread will not be able to know in a timely manner whether each operation has been completed, which does not meet the requirements of the always strategy. Therefore, the always strategy does not use a background sub-thread to execute fsync.

In addition, when using the AOF log, in order to prevent the log file from continuously increasing in size, Redis performs AOF rewriting to generate a new AOF log file with reduced volume. AOF rewriting itself takes a long time and is likely to block the Redis main thread. Therefore, Redis uses a child process to perform AOF rewriting.

However, there is a potential risk here: AOF rewriting will perform a large number of I/O operations on the disk, and at the same time, fsync needs to wait until the data is written to the disk before it can return. Therefore, when the pressure of AOF rewriting is high, it may cause fsync to be blocked. Although fsync is executed by a background sub-thread, the main thread monitors the progress of fsync execution.

When the main thread uses a background sub-thread to execute fsync, and when the new received operation record needs to be written back to the disk again, if the main thread finds that the previous fsync has not been completed, it will be blocked. Therefore, if the background sub-thread frequently blocks due to fsync execution (such as when AOF rewriting occupies a large amount of disk I/O bandwidth), the main thread will also be blocked, resulting in slower Redis performance.

To help you understand this, let me draw another picture to show the impact on the background sub-thread and the main thread when the disk pressure is low and high, respectively, after fsync.

Okay, now that we have reached this point, you have already understood that due to the existence of the background sub-thread for fsync and the child process for AOF rewriting, the main I/O thread is generally not blocked. However, if the amount of data written by the AOF rewriting child process is large during log rewriting, the fsync thread will be blocked, leading to the blocking of the main thread and increased latency. Now, let me give you some suggestions for troubleshooting and resolution.

Firstly, you can check the appendfsync configuration in the Redis configuration file, which indicates the AOF log writing strategy used by the Redis instance, as shown below:

If the AOF writing strategy is configured as everysec or always, please first confirm the data reliability requirements of the business side and determine whether every second or every operation needs to be logged. Some business users may not understand the Redis AOF mechanism and may directly use the highest level of data reliability (i.e., always) configuration. In fact, in some scenarios (such as Redis used for caching), it is acceptable to lose data because it can be retrieved from the backend database.

If the application is highly sensitive to latency but also allows a certain amount of data loss, you can set the no-appendfsync-on-rewrite configuration to yes, as shown below:

no-appendfsync-on-rewrite yes

When this configuration is set to yes, it means that no fsync operation is performed during AOF rewriting. This means that Redis instance can return directly without invoking a background thread for fsync after writing the write command to memory. However, if the instance crashes at this time, it will result in data loss. Conversely, if this configuration is set to no (which is the default configuration), fsync operation will still be performed by a background thread during AOF rewriting, which will cause blocking for the instance.

If you do need high performance while also requiring high data reliability, I suggest considering using a high-speed solid-state drive (SSD) as the write device for AOF log.

The bandwidth and concurrency of a high-speed SSD are more than ten times higher than those of traditional mechanical hard drives. When AOF rewriting and the fsync background thread are executed simultaneously, the SSD can provide sufficient disk I/O resources, reducing the competition for disk I/O resources between AOF rewriting and the fsync background thread, thereby reducing the impact on Redis performance.

Operating System: Swap #

If Redis’s AOF log configuration is set to no or if AOF mode is not used at all, will there be any other issues that could cause performance degradation?

Next, let’s discuss another potential bottleneck: the operating system’s memory swap.

Memory swap is a mechanism in the operating system that involves continuously moving data between memory and disk. Since it involves disk I/O, when swap is triggered, both the processes that have their data swapped in and the processes that have their data swapped out will experience performance degradation due to slower disk read and write operations.

Redis is an in-memory database that utilizes a large amount of memory. If memory usage is not properly controlled, or if it is running alongside other memory-intensive applications, Redis performance may be affected by swap, resulting in slower performance.

This is particularly important for Redis as it normally operates by directly accessing memory. When swap is triggered, Redis requests need to wait for disk I/O operations to complete. Furthermore, unlike the AOF log file read and write operations which use an fsync thread, swap negatively impacts the Redis main IO thread, significantly increasing Redis response time.

Speaking of this, let me share an example of performance degradation I once encountered due to swap.

Under normal circumstances, it would take 300 seconds for one of our running instances to complete 50 million GET requests. However, there was a time when it took nearly 4 hours to complete the same 50 million GET requests on this instance. After reproducing the issue, we discovered that swap had occurred on the machine where the instance was running. The latency increased nearly 48-fold, from 300 seconds to 4 hours, clearly demonstrating the severe impact of swap on performance.

So when does swap occur?

Typically, swap is triggered by insufficient physical machine memory. For Redis, there are two common scenarios:

  • Redis itself utilizes a large amount of memory, causing a shortage of available memory on the physical machine.
  • Other processes running on the same machine as Redis perform a significant number of file read and write operations. File I/O operations consume system memory, which reduces the amount of memory allocated to Redis and triggers swap.

For this problem, I also have a solution to suggest: increase the machine’s memory or use Redis clustering.

The operating system itself records the swap usage of each process in the background. You can start by checking the process ID of Redis using the following command, where 5332 is the process ID:

$ redis-cli info | grep process_id
process_id: 5332

Then, go to the directory of that process under the /proc directory of the machine where Redis is located:

$ cd /proc/5332

Finally, run the following command to check the usage of the Redis process. Here, I have only included part of the results:

$ cat smaps | egrep '^(Swap|Size)'
Size: 584 kB
Swap: 0 kB
Size: 4 kB
Swap: 4 kB
Size: 4 kB
Swap: 0 kB
Size: 462044 kB
Swap: 462008 kB
Size: 21392 kB
Swap: 0 kB

Each Size line represents the size of a memory block used by the Redis instance, and the corresponding Swap value denotes how much of that memory block has been swapped out to disk. If these two values are equal, it means the memory block has been completely swapped out to disk.

As an in-memory database, Redis uses many memory blocks of different sizes. Therefore, you can see multiple Size lines, some of which are very small (e.g., 4KB) while others are larger (e.g., 462044KB). The amount of each memory block that is swapped out to disk also varies. For example, in the previous results, the first 4KB memory block has been swapped out by 4KB, indicating that this block has been swapped. Additionally, the 462044KB memory block has been swapped out by 462008KB, which is approximately 462MB.

Here’s an important note: when swap becomes several hundred megabytes or even gigabytes in size, it indicates that the Redis instance is experiencing significant memory pressure and is likely to slow down. Therefore, the size of swap is an important indicator to investigate whether swap is causing the performance degradation of Redis.

Once memory swap occurs, the most direct solution is to increase the machine’s memory. If the instance is part of a Redis Shard Cluster, you could increase the number of Redis cluster instances to distribute the workload and reduce the memory requirement for each instance.

Of course, if the Redis instance shares the machine with other programs that perform intensive file operations (e.g., data analysis programs), you can migrate the Redis instance to a separate machine to meet its memory requirements. If this instance happens to be the master node in a Redis master-slave cluster, and the slave node has a large amount of memory, you can also consider switching the roles, making the high-memory slave node the new master to handle client requests.

Operating System: Transparent Huge Pages #

In addition to memory swapping, there is another factor related to memory that can affect Redis performance, which is the Transparent Huge Page (THP) mechanism.

The THP mechanism is supported by the Linux kernel starting from version 2.6.38. It allows for allocating memory pages in 2MB sizes, whereas regular memory page allocation is done in 4KB granularity.

Many people might think, “Redis is an in-memory database, so wouldn’t using huge pages be beneficial for Redis? Furthermore, won’t huge pages reduce the number of allocations when the same amount of memory is allocated? Isn’t it advantageous for Redis?”

In reality, the design of a system is usually a trade-off process, where advantages and disadvantages coexist. The use of huge pages by Redis is a typical example.

While huge pages can bring benefits in terms of memory allocation for Redis, let’s not forget that Redis needs to ensure data reliability by persistently storing data. This writing process is performed by additional threads, allowing the Redis main thread to continue accepting write requests from clients. These write requests from clients may modify the data that is currently being persisted. During this process, Redis adopts a copy-on-write mechanism, which means that when data needs to be modified, Redis doesn’t directly modify the data in memory, but instead creates a copy of the data and then performs the modification.

If huge pages are used, even if the client request only modifies 100B of data, Redis needs to copy the entire 2MB huge page. On the other hand, if regular memory pages are used, Redis only needs to copy 4KB. As you can see, when there are many client requests to modify or write new data, the use of huge pages will result in a large amount of copying, which will impact the normal memory access operations of Redis and eventually lead to slower performance.

So, what should we do? It’s simple, just disable huge pages.

First, we need to check if huge pages are enabled. You can do this by executing the following command on the machine running the Redis instance:

cat /sys/kernel/mm/transparent_hugepage/enabled

If the execution result is “always,” it means that huge pages are enabled. If it is “never,” it means that huge pages are disabled.

When deploying in a production environment, I recommend not using the huge page mechanism. Disabling it is also straightforward, just execute the following command:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Summary #

In this lesson, I introduced to you the methods of dealing with Redis slowness from two dimensions: the file system and the operating system.

To facilitate your application, I have compiled a checklist consisting of 9 checkpoints. I hope that when you encounter Redis performance issues, you can follow these steps to efficiently solve the problems.

  1. Obtain the baseline performance of the Redis instance in the current environment.
  2. Are slow query commands being used? If so, replace the slow query commands with other commands or perform aggregation calculations on the client side.
  3. Are the same expiration times set for expired keys? For bulk key deletion, you can add a random number to the expiration time of each key to avoid simultaneous deletion.
  4. Are there any bigkeys? For the deletion operation of bigkeys, if your Redis version is 4.0 or above, you can directly use asynchronous thread mechanism to reduce main thread blocking. If the Redis version is earlier than 4.0, you can use the SCAN command to iterate and delete. For collection queries and aggregation operations on bigkeys, you can use the SCAN command to complete them on the client side.
  5. What is the Redis AOF configuration level? Does the business really need this level of reliability? If we need high performance while allowing data loss, we can set the configuration item no-appendfsync-on-rewrite to “yes” to avoid AOF rewriting and fsync competing for disk IO resources, resulting in increased Redis latency. Of course, if both high performance and high reliability are needed, it is best to use high-speed solid-state disks as the write disk for AOF logs.
  6. Is the memory usage of the Redis instance too high? Has swapping occurred? If so, increase machine memory or use Redis cluster to distribute the number of key-value pairs and memory pressure of single-node Redis. At the same time, avoid sharing the machine with Redis and other memory-intensive applications.
  7. Has the transparent huge page mechanism been enabled in the running environment of the Redis instance? If so, simply disable the transparent huge page mechanism.
  8. Is a Redis master-slave cluster running? If so, control the data size of the master instance between 2 to 4GB to prevent the slave from blocking due to loading large RDB files during replication.
  9. Are multiple CPU cores or NUMA architecture machines used to run Redis instances? When using multiple CPU cores, you can bind the Redis instance to physical cores; when using NUMA architecture, pay attention to running the Redis instance and network interruption handling program on the same CPU socket.

In fact, there are many factors that affect system performance. The solutions discussed in these two lessons are for dealing with the most common problems.

If you encounter special situations, don’t panic. I will share another tip with you: carefully check if there are annoying “neighbors”, specifically speaking, whether there are other programs on the machine where Redis is located that consume memory, disk IO, and network IO, such as database programs or data collection programs. If there are, I suggest you migrate these programs to other machines.

In order to ensure high performance of Redis, we need to provide Redis with sufficient computing, memory, and IO resources, and provide it with a “quiet” environment.

One question per lesson #

In these two sessions, I introduced to you the methods of systematically identifying, troubleshooting, and solving Redis performance issues. So, I would like to ask, have you ever encountered a situation where Redis was slow? If so, how did you solve it?

Please feel free to share your experiences in the comments section. If you found today’s content helpful, please also consider sharing it with your friends or colleagues. See you in the next lesson.