21 Buffer Overflow a Place That Could Trigger Disasters

21 Buffer Overflow A Place That Could Trigger Disasters #

Today, let’s learn about the usage of buffers in Redis.

The functionality of a buffer is actually quite simple. Its main purpose is to temporarily store command data in a block of memory, in order to prevent data loss and performance issues caused by the processing speed of data and commands being slower than the sending speed. However, due to the limited memory space of the buffer, if the speed of writing data into it continues to be higher than the speed of reading data from it, it will result in the buffer needing more and more memory to temporarily store the data. When the memory occupied by the buffer exceeds the set upper threshold, a buffer overflow will occur.

If an overflow occurs, data will be lost. Does that mean not setting a limit on the size of the buffer will solve the problem? Obviously not. As more and more data accumulates, the buffer will occupy more and more memory space. Once the available memory of the machine where the Redis instance is located is exhausted, it will cause the Redis instance to crash.

So it’s no exaggeration to say that the buffer is used to prevent the disaster of request or data loss. But only by using it correctly can it truly serve its “prevention” purpose.

As we know, Redis is a typical client-server architecture, and all operation commands need to be sent to the server side through the client. Therefore, one of the main application scenarios of the buffer in Redis is to temporarily store the command data sent by the client or the data results returned by the server between the client and server. In addition, another major application scenario of the buffer is to temporarily store the write commands and data received by the main node during data synchronization between master and slave nodes.

In this lesson, we will discuss the buffer overflow issues and corresponding solutions separately for the server side and the client side, as well as for the master-slave cluster.

Client Input and Output Buffers #

Let’s first take a look at the buffers between the server and the client.

In order to avoid mismatched speeds between the client and server in sending and processing requests, the server sets up an input buffer and an output buffer for each connected client, which we call the client input buffer and the client output buffer.

The input buffer temporarily stores the commands sent by the client, and the Redis main thread reads the commands from the input buffer for processing. After the Redis main thread has processed the data, it writes the results to the output buffer and returns them to the client through the output buffer, as shown in the following figure:

Next, let’s learn about the cases of input buffer overflow and output buffer overflow, as well as the corresponding solutions.

How to handle input buffer overflow? #

As we have analyzed before, the input buffer is used to temporarily store the commands sent by the client, so there are mainly two situations that can cause overflow:

Writing a big key, such as writing multiple millions of set-type data at once;
The server processes the requests too slowly, for example, the Redis main thread encounters intermittent blocking and cannot process the sent requests in a timely manner, resulting in the accumulation of requests sent by the client in the buffer.

Next, we will continue learning from the perspective of how to check the memory usage of the input buffer and how to avoid overflow.

To check the usage of the input buffer for each client connected to the server, we can use the CLIENT LIST command:

CLIENT LIST
id=5 addr=127.0.0.1:50487 fd=9 name= age=4 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=32742 obl=0 oll=0 omem=0 events=r cmd=client

Although the output of the CLIENT command contains a lot of information, we only need to focus on two types of information.

One is the information of the connected clients to the server. This example shows the situation of the input buffer for a client, and if there are multiple clients, the addr in the output will display the IP and port number of different clients.

The other type is the three parameters related to the input buffer:

cmd, which represents the latest command executed by the client. In this example, the executed command is CLIENT.
qbuf, which represents the amount of input buffer already used. In this example, the CLIENT command has used a buffer of 26 bytes.
qbuf-free, which represents the amount of input buffer not yet used. In this example, the CLIENT command can still use a buffer of 32742 bytes. The sum of qbuf and qbuf-free is the total size of the buffer allocated by the Redis server for this connected client. In this example, a total of 26 + 32742 = 32768 bytes, which is a buffer of 32KB, is allocated.

With the CLIENT LIST command, we can judge the memory usage of the client input buffer based on the output. If qbuf is large, but qbuf-free is small, we need to pay attention because it means that the input buffer has already occupied a lot of memory and there is little free space. At this time, if the client writes a large number of commands, it will cause overflow of the input buffer of the client and Redis will respond by closing the client connection, resulting in the inability to access data in the business program.

Normally, Redis servers do not serve only one client. When the total memory occupied by multiple client connections exceeds the maxmemory configuration of Redis (e.g. 4GB), Redis will trigger data eviction. Once data is evicted from Redis, if it needs to be accessed again, it needs to be read from the backend database, which reduces the access performance of the business application. Moreover, even worse, if using multiple clients causes Redis to occupy too much memory, it can lead to out-of-memory problems and eventually cause Redis to crash, seriously affecting the business application.

Therefore, we must find ways to avoid input buffer overflow. We can consider two aspects: increasing the buffer size and improving the speed of sending and processing data commands.

Firstly, let’s see if there is a way to adjust the size of the input buffer through parameters. No, there isn’t.

The upper limit threshold for the size of the Redis client input buffer is set to 1GB in the code. In other words, Redis server allows storing a maximum of 1GB of commands and data for each client. 1GB is already suitable for most production environments. On the one hand, this size is sufficient for processing the requests of the vast majority of clients; on the other hand, if it is larger, Redis may crash due to excessive memory resources occupied by clients.

Therefore, Redis does not provide parameters to adjust the size of the client input buffer. If we want to avoid input buffer overflow, we can only start with the speed of sending and processing data commands, that is, the aforementioned methods of avoiding the client from writing big keys and preventing the Redis main thread from blocking.

Next, let’s take a look at the issue of output buffer overflow.

How to handle output buffer overflow? #

The output buffer of Redis stores the data that the Redis main thread wants to return to the client. Generally, the data returned to the client by the main thread of Redis includes simple and fixed-size OK responses (e.g. executing SET command) or error messages, as well as variable-sized response results that include specific data (e.g. executing HGET command).

Therefore, the output buffer set up by Redis for each client also includes two parts: one is a fixed buffer space of 16KB used to store OK responses and error messages, and the other is a dynamically increasing buffer space used to store variable-sized response results.

Under what circumstances will there be an output buffer overflow? I have summarized three situations for you:

The server returns a large number of results for bigkey.
The MONITOR command is executed.
The buffer size is set improperly.

Among them, bigkey originally takes up a large amount of memory space, so the results returned by the server will include bigkey, which will inevitably affect the output buffer. Next, let’s focus on the execution of the MONITOR command and the setting of the buffer size.

The MONITOR command is used to monitor the execution of Redis. After executing this command, it will continuously output the monitored command operations, as shown below:

MONITOR
OK
1600617456.437129 [0 127.0.0.1:50487] "COMMAND"
1600617477.289667 [0 127.0.0.1:50487] "info" "memory"

By now, have you noticed any issues? The output result of MONITOR will continuously occupy the output buffer, and the occupation will continue to increase, resulting in overflow. So, here’s a suggestion: MONITOR command is primarily used in debugging environments and should not be continuously used in production environments. Of course, it’s fine to occasionally use MONITOR to check Redis command execution in a production environment.

Next, let’s look at the issue of setting the output buffer size. We need to take a look at the issue of setting the output buffer size. Unlike the input buffer, we can set the size of the buffer through the client-output-buffer-limit configuration option. The specific settings include two aspects:

Setting the upper threshold value for the buffer size.
Setting the upper threshold value for the continuous writing data volume and the upper threshold value for the continuous writing data time.

When using client-output-buffer-limit to set the buffer size, we need to distinguish the types of clients.

For applications that interact with Redis instances, there are two main types of clients that interact with the Redis server: the normal client that reads and writes commands interactively with the Redis server, and the subscriber client that subscribes to Redis channels. In addition, in Redis master-slave clusters, there is also a type of client on the master node (slave node client) used to synchronize data with the slave node, and I will explain the buffer in master-slave clusters in more detail.

When setting the buffer size for a normal client, it is common to make this setting in the Redis configuration file:

client-output-buffer-limit normal 0 0 0

Among them, normal indicates that the current setting is for a normal client. The first 0 sets the buffer size limit, and the second and third 0s represent the limits for continuous writing volume and continuous writing time of the buffer respectively.

For a normal client, after sending a request, it waits for the request result to be returned before sending the next request. This sending method is called blocking sending. In this case, unless reading a bigkey with a particularly large volume, the output buffer of the server-side is generally not blocked.

Therefore, we usually set the buffer size limit, as well as the limits for continuous writing volume and continuous writing time of the normal client to 0, which means no restrictions are set.

For a subscriber client, once a subscribed Redis channel has a message, the server-side will send the message to the client through the output buffer. Therefore, the message sending method between subscriber clients and the server does not belong to blocking sending. However, if there are many channel messages, it will also occupy a lot of output buffer space.

Therefore, we will set the buffer size limit, the limit for continuous writing volume, and the limit for continuous writing time for the subscriber client. You can set it in the Redis configuration file as follows:

client-output-buffer-limit pubsub 8mb 2mb 60

Among them, pubsub indicates that the current setting is for the subscriber client. 8mb indicates the maximum size of the output buffer is 8MB. If the actual occupied buffer size exceeds 8MB, the server will directly close the client’s connection. The 2mb and 60 indicate that if the writing volume of the output buffer exceeds 2MB continuously within 60 seconds, the server will also close the client’s connection.

Alright, let’s summarize how to deal with output buffer overflow:

Avoid operations that return a large amount of data for bigkey.
Avoid continuous use of the MONITOR command in a production environment.
Use client-output-buffer-limit to set a reasonable upper limit for the buffer size or limits for continuous writing time and volume.

The above is the important content we need to master about client-side buffers. Let’s continue to see what issues we need to pay attention to when using buffers between master and slave clusters.

Buffer in Master-Slave Cluster #

Data replication between master and slave clusters includes two types: full replication and incremental replication. Full replication synchronizes all data, while incremental replication only synchronizes the commands received by the master during the network disconnection period with the slave. Regardless of the type of replication, a buffer is used to ensure data consistency between master and slave nodes. However, the buffer used in these two replication scenarios is different in terms of overflow impact and size settings. So, let’s study them separately.

Overflow Problem of Replication Buffer #

During full replication, while the master node is transferring the RDB file to the slave node, it continues to receive write command requests from the client. These write commands are first saved in the replication buffer and then sent to the slave node to execute after the RDB file transfer is completed. The master node maintains a replication buffer for each slave node to ensure data synchronization between the master and slave nodes.

Therefore, if the slave node is slow in receiving and loading the RDB file during full replication, and at the same time, the master node receives a large number of write commands, the write commands in the replication buffer will accumulate and eventually overflow.

In fact, the replication buffer on the master node is essentially an output buffer used for the client (called the slave client) connected to the slave node. Once the replication buffer overflows, the master node will directly close the connection for replication with the slave node, causing full replication to fail. So how can we avoid the overflow of the replication buffer?

On the one hand, we can control the amount of data stored on the master node. According to common usage experience, we usually control the data size on the master node to be between 2 and 4 GB, which can make the full synchronization process faster and avoid excessive accumulation of commands in the replication buffer.

On the other hand, we can use the client-output-buffer-limit configuration option to set a reasonable size for the replication buffer. The setting is based on the size of the data on the master node, the write load pressure on the master node, and the memory size of the master node.

Let’s learn how to set it with a specific example. Execute the following command on the master node:

config set client-output-buffer-limit slave 512mb 128mb 60

In this command, the slave parameter indicates that this configuration item is for the replication buffer. 512mb sets the upper limit of the buffer size to 512 MB, and 128mb and 60 represent the following settings: if the write volume exceeds 128 MB within 60 seconds, it will also trigger buffer overflow.

Let’s see what this setting is useful for. Suppose a write command data is 1 KB, then the replication buffer can accumulate up to 512K commands (512 MB / 1 KB = 512K) in it. At the same time, the maximum write command rate that the master node can tolerate during full replication is 2000 commands/s (128 MB / 1 KB / 60 is approximately equal to 2000).

In this way, we come up with a method: when setting the size of the replication buffer in actual applications, we can roughly estimate the amount of accumulated write command data in the buffer based on the size of the write command data and the actual load of the application (i.e., the write command rate); then, compare it with the size of the replication buffer set to determine whether the buffer size set is sufficient to support the accumulated write command data.

Regarding the replication buffer, we will encounter another problem. The memory overhead of the replication buffer on the master node will be the sum of the occupied memory of the output buffer of each slave client connected to the master node. If there are a large number of slave nodes in the cluster, the memory overhead on the master node will be very large. Therefore, we must also control the number of slave nodes connected to the master node and avoid using a large-scale master-slave cluster.

Okay, let’s summarize the content of this part. To avoid the overflow of the replication buffer and the resulting failure of full replication due to the accumulation of too many commands, we can control the amount of data stored on the master node and set a reasonable size for the replication buffer. At the same time, we need to control the number of slave nodes to avoid excessive memory usage of the replication buffer on the master node.

Overflow Problem of Replication Backlog Buffer #

Next, let’s take a look at the buffer used in incremental replication, which is called the replication backlog buffer.

When the master node synchronizes the received write commands to the slave nodes, it also writes these write commands to the replication backlog buffer. Once the slave node experiences a network interruption and reconnects to the master node, it will read the write commands received by the master during the disconnection period from the replication backlog buffer, and then perform incremental synchronization, as shown in the following diagram:

Does this look familiar to you? Yes, we have already learned about the replication backlog buffer in [Lesson 6], but at that time, I told you its English name was repl_backlog_buffer. So, in this lesson, let’s review two key points from the perspective of buffer overflow: the impact of replication backlog buffer overflow and how to deal with the overflow problem of the replication backlog buffer.

First, the replication backlog buffer is a circular buffer with a limited size. When the replication backlog buffer on the master node is full, it will overwrite the older command data in the buffer. If the slave node has not synchronized these older command data, it will cause the master and slave nodes to restart full replication.

Second, to address the overflow problem of the replication backlog buffer, we can adjust the size of the replication backlog buffer by setting the value of the repl_backlog_size parameter. The specific adjustment basis can be found in the calculation basis of the repl_backlog_size in [Lesson 6].

Summary #

In this lesson, we have learned about the use of buffers in Redis. By using buffers, we can avoid losing command data when the receiving end cannot keep up with the sending speed of the sender.

Based on the purpose of the buffer, such as for client communication or for master-slave replication, I have divided the buffer into input and output buffers for clients, as well as replication and backlog buffers on the master node in a master-slave cluster. The benefit of this approach is that you can clearly see where buffers are used in Redis. This allows you to quickly identify the direction to investigate when troubleshooting issues, by analyzing the communication process between clients and the server, as well as the replication process between master and slave nodes.

Now, let’s summarize the four buffers based on the impact of buffer overflow on Redis.

Buffer overflow leading to network connection closure: The buffers used by regular clients, subscription clients, and slave clients are essentially used to transmit command data between Redis clients and servers, or between master and slave nodes. Once these buffers overflow, the handling mechanism is to directly close the connection between the client and the server, or between master and slave nodes. The direct impact of network connection closure is that the business program cannot read or write to Redis, or the full synchronization between master and slave nodes fails and needs to be re-executed.
Buffer overflow leading to loss of command data: The backlog buffer for replication on the master node is a circular buffer. Once it overflows, new write command data will overwrite old command data, causing the loss of old command data, and resulting in a full replication between master and slave nodes.

Essentially, buffer overflow can be caused by three reasons: command data being sent too quickly or being too large, command data being processed slowly, or the buffer space being too small. Once you understand this, you can take targeted countermeasures.

To address the issue of command data being sent too quickly or being too large, for regular clients, you can avoid big keys, and for the replication buffer, it means avoiding overly large RDB files.
To address the issue of command data being processed slowly, the solution is to reduce blocking operations on the Redis main thread, such as using asynchronous delete operations.
To address the issue of the buffer space being too small, the solution is to use the client-output-buffer-limit configuration to set reasonable sizes for the output buffer, replication buffer, and backlog buffer. However, we should not forget that the size of the input buffer is fixed by default and cannot be modified through configuration unless the Redis source code is directly modified.

With these countermeasures, I believe you will be able to avoid the loss of command data and the “disaster” of Redis crashing due to buffer overflow in practical applications.

One Question per Lesson #

Finally, let me ask you a small question.

In this lesson, we mentioned that Redis adopts a client-server architecture, where the server side maintains input and output buffers for each client. So, when the application program interacts with the Redis instance, does the client used in the application program need to use buffers? If so, will it have any impact on Redis’ performance and memory usage?

Please feel free to write down your thoughts and answers in the comment section. Let’s communicate and discuss together. If you find today’s content helpful, you are also welcome to share it with your friends or colleagues. See you in the next lesson.