32 Sync and Failure Switching in Redis Master Slave What Are the Pitfalls

32 Sync and Failure Switching in Redis Master-Slave What are the Pitfalls #

Redis’s master-slave synchronization mechanism not only allows the slave servers to handle more read requests and relieve the pressure on the master server, but also enables failover from the master to the slave in the event of a master failure, providing highly reliable service.

However, when using the master-slave mechanism in practice, we can easily fall into some pitfalls. In this lesson, I will introduce three pitfalls: inconsistent master-slave data, reading expired data, and improper configuration settings that lead to service failure.

Once we encounter these pitfalls, not only can the business application read incorrect data, but it can also cause Redis to become unusable. Therefore, we must fully understand the causes of these pitfalls and prepare a set of avoidance plans in advance. However, even if we accidentally fall into the trap, there is no need to worry, as I will also introduce the corresponding solutions.

Alright, without further ado, let’s start with the first pitfall: inconsistent master-slave data.

Inconsistent Data in Master-Slave Replication #

Inconsistent data in master-slave replication refers to the situation where the value read by the client from the slave database is not the same as the latest value in the master database.

For example, let’s say the age of a user in the master-slave databases was previously stored as 19. However, the master database received a modification command and updated this data to 20. Despite this update, the value in the slave database remains 19. As a result, if the client reads the user’s age from the slave database, they will get the old value.

So why does this discrepancy occur? It is because the command replication between the master and slave databases is asynchronous.

Specifically, during the command propagation phase between the master and slave databases, when the master database receives a new write command, it sends it to the slave database. However, the master database does not wait for the slave database to actually execute the command and return the result to the client. Instead, the master database returns the result to the client after executing the command locally. If the slave database has not executed the command synchronized from the master database, there will be data inconsistency between the master and slave databases.

Under what circumstances does the slave database lag behind in executing synchronization commands? There are primarily two reasons.

Firstly, there may be network transmission delays between the master and slave databases, preventing the slave database from promptly receiving the commands from the master database, thus delaying the execution of synchronization commands on the slave database.

Secondly, even if the slave database promptly receives the commands from the master database, it may still be blocked due to the execution of other complex commands (such as set operations). In this case, the slave database needs to finish executing the current command before executing the command sent by the master database, resulting in data inconsistency between the master and slave databases. Furthermore, during the time when the execution of commands on the master database is delayed, the master database itself may execute new write operations. This further exacerbates the inconsistency between the master and slave databases.

So, how should we deal with this? I will provide you with two methods.

First, in terms of hardware configuration, we should try to ensure a good network connection between the master and slave databases. For example, we should avoid deploying the master and slave databases in different data centers, or avoid deploying network-intensive applications (such as data analysis applications) together with Redis master-slave databases.

Additionally, we can develop an external program to monitor the replication progress between the master and slave databases.

Since the Redis command INFO replication can be used to view the progress of the master database receiving write commands (master_repl_offset) and the progress of the slave database replicating write commands (slave_repl_offset), we can develop a monitoring program that first uses the INFO replication command to obtain the progress of the master and slave databases. Then, by subtracting slave_repl_offset from master_repl_offset, we can calculate the replication progress difference between the slave and master databases.

If the progress difference for a certain slave database exceeds a preset threshold, we can instruct the client not to connect to that slave database for data reading, thereby reducing the occurrence of reading inconsistent data. However, to avoid situations where the client cannot connect to any slave database, the threshold for the replication progress difference needs to be set larger.

When using Redis in applications, we can periodically run this process to monitor the inconsistency between the master and slave databases. To help you better understand this method, I have created a flowchart for you to refer to.

Of course, the monitoring program can continuously monitor the replication progress of the slave databases. When the replication progress of a slave database catches up with the master database, we can allow the client to connect to these slave databases again.

In addition to inconsistent data in master-slave replication, sometimes we may also read expired data in the slave database. Let’s analyze this in detail next.

Reading expired data #

Sometimes, when using a Redis master-slave cluster, we might read expired data. For example, if the expiration time of data X is 202010240900, the client can still read data X from the slave at 202010240910. Once data has expired, it should be deleted, and the client should not be able to read that data anymore. However, why can Redis still read expired data from the slave?

This is actually caused by Redis’ expiration data deletion policy. Let me explain it to you in detail.

Redis uses two strategies to delete expired data, which are lazy deletion and periodic deletion.

Let’s start with the lazy deletion strategy. When the expiration time of data is reached, it is not immediately deleted. Instead, it is checked for expiration when there is a request to read or write this data. If the data is found to be expired, it is deleted at that time.

The advantage of this strategy is that it minimizes the use of CPU resources for deletion operations. If a data is not needed, there is no need to waste time checking and deleting it. However, this strategy can result in a large number of expired data being kept in memory, occupying a significant amount of memory resources. Therefore, while Redis uses this strategy, it also uses a second strategy: periodic deletion.

Periodic deletion refers to Redis randomly selecting a certain number of data at regular intervals (default 100ms), checking if they have expired, and deleting the expired data to release memory.

Now that we understand these two deletion strategies, let’s take a look at why we can read expired data.

First of all, although periodic deletion can release some memory, Redis does not check a large number of data at each deletion operation to avoid performance impact. If there are many expired data that have not been accessed, these data will remain in the Redis instance. This is a significant factor in why business applications can read expired data - the existence of these remaining data.

Secondly, with the implementation of lazy deletion strategy, the data is only actually deleted when it is accessed again. If the client reads the remaining expired data from the master, the master triggers the deletion operation, and the client does not read the expired data. However, the slave does not execute the deletion operation itself. If the client accesses the remaining expired data on the slave, the slave does not trigger the deletion of data. So, will the slave return expired data to the client?

This depends on the version of Redis you are using. If you are using a version earlier than Redis 3.2, the slave, when serving read requests, does not check whether the data has expired and returns expired data. Since version 3.2, Redis has made improvements. If the data being read is already expired, the slave does not delete it but returns a null value, thus avoiding the client reading expired data. Therefore, when using a master-slave cluster, it is recommended to use Redis 3.2 or higher.

You may wonder, if I use Redis version 3.2 or higher, will I never read expired data again? Actually, you still can.

Why is this happening? It is related to the Redis commands used to set expiration time. Some commands may delay the expiration time on the slave, resulting in the expired data being read from the slave. Let me explain it to you in detail.

First, let me introduce these commands to you. There are a total of 4 commands used to set data expiration time, and we can divide them into two categories:

  • EXPIRE and PEXPIRE: They set the time-to-live (TTL) of data starting from the execution of the command.
  • EXPIREAT and PEXPIREAT: They directly set the expiration time of data to a specific point in time.

The parameters and meanings of these 4 commands are shown in the following table:

To help you understand, let me give you two examples.

The first example is to use the EXPIRE command. When you execute the following command, the expiration time of “testkey” will be set to 60 seconds later.

EXPIRE testkey 60

The second example is to use the EXPIREAT command. For example, when you execute the following command, it will make “testkey” expire at 9:00 AM on October 24, 2020. The value “1603501200” in the command represents the timestamp of 9:00 AM in seconds.

EXPIREAT testkey 1603501200

Now that you know these commands, let’s take a look at how they can lead to reading expired data.

When the master-slave replication is happening, if the master receives an EXPIRE command, it will execute it directly. This command will be sent to the slave to be executed after the full replication is completed. When the slave executes it, it will add the remaining time of the data’s expiration based on the current time. As a result, the expiration time of the data on the slave will be delayed compared to the master.

This may be a bit hard to understand, so let me give you another example.

Suppose the current time is 9:00 AM on October 24, 2020, and the master and slave are synchronizing. The master receives a command: EXPIRE testkey 60. This means that the expiration time of “testkey” on the master is 9:01 AM. The master directly executes this command.

However, it takes 2 minutes for the full replication between the master and slave to complete. When the slave starts executing this command, the time is already 9:02 AM. The EXPIRE command sets the expiration time of “testkey” to 60 seconds after the current time, which is 9:03 AM. If a client reads “testkey” on the slave at 9:02:30 AM, it can still get the value of “testkey”. However, “testkey” has actually expired.

To avoid this situation, my suggestion to you is to use the EXPIREAT/PEXPIREAT command in your business application to set the expiration time to a specific point in time, thus avoiding reading expired data.

Now, let’s summarize the two typical pitfalls we just learned.

  • Inconsistent data between master and slave. Redis adopts asynchronous replication, so it is difficult to achieve strong consistency guarantee (keeping master and slave data in sync all the time), and data inconsistency is hard to avoid. I provided you with a solution: ensure a good network environment and use program monitoring to check the replication progress of the slaves. Once the replication progress exceeds a threshold, do not let clients connect to the slaves.

  • Reading expired data can be proactively prevented. One method is to use Redis 3.2 or above. Also, you can use the EXPIREAT/PEXPIREAT command to set expiration time, avoiding the delayed expiration time on the slaves. However, one thing to note here is that since EXPIREAT/PEXPIREAT sets a specific point in time, the clocks of the master and slave nodes must be synchronized. The specific approach is to synchronize the clocks of the master and slave nodes with the same NTP server (time server).

Apart from pitfalls in the synchronization process, there are also pitfalls when master-slave failover occurs due to improper configuration. Next, let me introduce two scenarios where services go down due to improper configuration options.

Service Crash Caused by Unreasonable Configuration Items #

There are two configuration items involved here, namely protected-mode and cluster-node-timeout.

  1. The protected-mode Configuration Item

This configuration item determines whether the sentinel instance can be accessed by other servers. When this configuration item is set to yes, the sentinel instance can only be accessed locally on the deployed server. When set to no, other servers can also access the sentinel instance.

Because of this, if protected-mode is set to yes and the other sentinel instances are deployed on other servers, these sentinel instances cannot communicate with each other. When the master fails, the sentinel cannot determine that the master is offline and cannot perform master-slave switch, resulting in Redis service becoming unavailable.

Therefore, when applying the master-slave cluster, we should pay attention to setting the protected-mode configuration item to no and set the bind configuration item to the IP addresses of other sentinel instances. This ensures that only sentinels with IP addresses set in bind can access the current instance, ensuring communication between instances for master-slave switch and the security of the sentinels.

Let’s look at a simple example. If the following configuration items are set, the sentinel instances deployed on servers 192.168.10.3/4/5 can communicate with each other and perform master-slave switch:

protected-mode no
bind 192.168.10.3 192.168.10.4 192.168.10.5
  1. The cluster-node-timeout Configuration Item

This configuration item sets the timeout for the instance to respond to heartbeat messages in Redis Cluster.

When we configure a “one master one slave” mode for each instance in Redis Cluster, if the master instance fails, the slave instance will switch to become the master. Due to network latency and the time required for the switch operation, the switch time may be long, resulting in the instance’s heartbeat timing out (exceeding cluster-node-timeout). When the instance times out, Redis Cluster considers it abnormal. The normal operation condition for Redis Cluster is that more than half of the instances can operate normally.

Therefore, if the number of instances performing master-slave switch exceeds half and the switch time is too long, it is possible that more than half of the instances will have heartbeat timeouts, which may cause the entire cluster to crash. So, I recommend increasing the cluster-node-timeout (for example, to 10 to 20 seconds).

Summary #

In this lesson, we learned about three potential pitfalls that may occur during master-slave synchronization in Redis. They include inconsistent data between the master and slave, reading expired data, and server crashes due to improper configuration.

To help you understand more easily, I have summarized the causes and solutions for these pitfalls in the table below, which you can review.

Finally, regarding the issue of inconsistent data between the master and slave, I would like to give you another suggestion: the configuration item slave-serve-stale-data in Redis determines if the slave can handle read and write commands. You can set it to no. This way, the slave can only serve INFO and SLAVEOF commands, which can prevent reading inconsistent data from the slave.

However, please be aware of the difference between this configuration item and slave-read-only. slave-read-only determines if the slave can handle write commands. When set to yes, the slave can only handle read requests and cannot handle write requests, so be careful not to confuse them.

A question for each lesson #

As usual, I have a little question for you. We set slave-read-only to no, allowing the slave database to directly delete data. This is done to avoid reading outdated data. Do you think this is a good method?

Feel free to share your thoughts and answers in the comments. Let’s discuss and exchange ideas together. If you found today’s content helpful, feel free to share it with your friends or colleagues. See you in the next lesson.