08 Sentinel Cluster What Happens to Master-Slave Database When Sentinel Goes Down #

In the previous lesson, we learned about the Sentinel mechanism, which can achieve automatic switching between master and slave databases. By deploying multiple instances, a Sentinel cluster is formed. Multiple instances in the Sentinel cluster work together to reduce the misjudgment rate of the master database going offline.

However, we still need to consider one question: if a Sentinel instance fails during runtime, can the master-slave switch still work?

In fact, once multiple instances form a Sentinel cluster, even if a Sentinel instance fails and goes down, other Sentinels can continue to work together to complete the master-slave switch, including determining whether the master database is offline, selecting a new master database, and notifying the slave databases and clients.

If you have deployed a Sentinel cluster, you would know that when configuring the Sentinel’s information, we only need to use the following configuration item to set the IP and port of the master database, without configuring the connection information of other Sentinels.

sentinel monitor <master-name> <ip> <redis-port> <quorum>

Since these Sentinel instances are unaware of each other’s addresses, how do they form a cluster? To understand this question, we need to learn about the composition and operation mechanism of a Sentinel cluster.

Sentinel Cluster Composition Based on Pub/Sub Mechanism #

Sentinel instances can discover each other, thanks to the pub/sub mechanism provided by Redis, also known as the publish/subscribe mechanism.

Once a Sentinel instance establishes a connection with the master database, it can publish messages on the master database, such as its own connection information (IP and port). At the same time, it can also subscribe to messages from the master database to obtain connection information published by other Sentinel instances. When multiple Sentinel instances perform publish and subscribe operations on the master database, they can learn each other’s IP addresses and ports.

In addition to Sentinel instances, the applications we write can also use Redis for message publishing and subscription. Therefore, in order to differentiate messages from different applications, Redis manages these messages in channels. The so-called channel is actually the category of messages. Messages of the same category belong to the same channel. Otherwise, they belong to different channels. Only applications that subscribe to the same channel can exchange information through published messages.

In a master-slave cluster, there is a channel called “sentinel:hello” on the master database, through which Sentinels discover each other and communicate with each other.

Let me give you an example to explain in detail. In the diagram below, Sentinel 1 publishes its IP address (172.16.19.3) and port (26579) to the “sentinel:hello” channel, and Sentinel 2 and 3 subscribe to this channel. At this point, Sentinel 2 and 3 can directly obtain the IP address and port number of Sentinel 1 from this channel.

Then, Sentinel 2 and 3 can establish a network connection with Sentinel 1. In this way, Sentinel 2 and 3 can also establish network connections with each other, and thus the Sentinel cluster is formed. They can communicate with each other through network connections, for example, to determine and negotiate whether the master database is offline.

In addition to establishing connections among themselves to form a cluster, Sentinels also need to establish connections with the slave databases. This is because in the monitoring tasks of Sentinels, they need to monitor both the master and slave databases, and after a master-slave switch is completed, they need to notify the slave databases to synchronize with the new master database.

So, how does Sentinel know the IP addresses and ports of the slave databases?

This is accomplished by Sentinel sending the INFO command to the master database. As shown in the following diagram, Sentinel 2 sends the INFO command to the master database, and the master database returns the list of slave databases to Sentinel as a response to this command. Then, based on the connection information in the slave database list, Sentinel can establish connections with each slave database and continuously monitor them through these connections. Sentinel 1 and 3 can establish connections with the slave databases in the same way.

You see, through the pub/sub mechanism, Sentinels can form a cluster, and Sentinels can also obtain the connection information of the slave databases and establish connections with them for monitoring by using the INFO command.

However, Sentinels cannot only connect to the master and slave databases. After a master-slave switch, clients also need to know the connection information of the new master database in order to send requests to it. Therefore, Sentinels also need to complete the task of informing clients of the information of the new master database.

Moreover, when using Sentinels in practice, we sometimes encounter the issue of how clients can monitor the process of Sentinel performing master-slave switches. For example, which step is the master-slave switch currently at? This actually requires clients to obtain various events that occur during the monitoring, leader selection, and switch process of the Sentinel cluster.

At this time, we can still rely on the pub/sub mechanism to help us achieve information synchronization between Sentinels and clients.

Client Event Notification Based on the Pub/Sub Mechanism #

Essentially, a sentinel is a Redis instance running in a specific mode. However, it does not serve request operations. Its tasks include monitoring, leader selection, and notification. Therefore, each sentinel instance also provides the pub/sub mechanism, allowing clients to subscribe to messages from sentinels. There are many channels available for subscription in sentinels, and different channels contain different key events that occur during the process of master-slave switching.

With so many channels, it’s easy to lose focus when trying to learn all of them at once. To alleviate your learning pressure, I have summarized the important channels that involve several key events, including the determination of the offline status of the master, selection of a new master, and reconfiguration of the slave.

After knowing these channels, you can let the client subscribe to the messages from the sentinels. The specific steps are as follows: after reading the configuration file of the sentinel, the client can obtain the address and port of the sentinel and establish a network connection with it. Then, we can execute the SUBSCRIBE command in the client to retrieve different event messages.

For example, you can execute the following command to subscribe to the “events indicating all instances entering the objective offline state”:

SUBSCRIBE +odown

Of course, you can also execute the following command to subscribe to all events:

PSUBSCRIBE  *

Once the sentinel has selected a new master, the client will see the switch-master event. This event indicates that the master has been switched and the IP address and port information of the new master are available. At this point, the client can use the IP address and port of the new master to communicate with it.

switch-master <master name> <oldip> <oldport> <newip> <newport>

With these event notifications, the client can not only obtain the connection information of the new master after a master-slave switch, but also monitor various important events that occur during the process of master-slave switching. In this way, the client can know at which step the switch is, which is helpful for understanding the progress of the switch.

Great! With the pub/sub mechanism, connections can be established between sentinels, sentinels and slaves, and sentinels and clients. Combined with the last lesson where we introduced the determination of master’s offline state and the basis for leader selection, the monitoring, leader selection, and notification tasks of the sentinel cluster can work properly. However, we still need to consider one question: after the master fails, if there are multiple sentinel instances in the sentinel cluster, how do we determine which sentinel will actually perform the master-slave switch?

Which sentinel performs master-slave switch? #

The process of determining which sentinel performs the master-slave switch is similar to the process of determining “objective offline” of the primary database. It is also a process of “voting arbitration”. Before understanding this process in detail, let’s take a look at the arbitration process of determining “objective offline”.

In order for the sentinel cluster to determine that the primary database is “objectively offline”, a certain number of instances in the cluster need to consider that the primary database has already been “subjectively offline”. In the previous lesson, I introduced the principles of determining “objective offline”. Next, I will explain the specific process.

Once any instance judges that the primary database is “subjectively offline”, it will send the “is-master-down-by-addr” command to other instances. Then, based on their own connection status with the primary database, other instances will respond with Y or N, where Y represents an affirmative vote and N represents a dissenting vote.

Once a sentinel obtains the required number of affirmative votes through arbitration, it can mark the primary database as “objectively offline”. The required number of affirmative votes is set by the “quorum” configuration in the sentinel’s configuration file. For example, if there are 5 sentinels and the “quorum” is configured as 3, then a sentinel only needs to obtain 3 affirmative votes to mark the primary database as “objectively offline”. These 3 affirmative votes include one affirmative vote from the sentinel itself and two affirmative votes from other sentinels.

At this point, the sentinel can send commands to other sentinels indicating that it wants to perform the master-slave switch and ask all other sentinels to vote. This voting process is called “Leader election”. Because the sentinel that ultimately performs the master-slave switch is called the Leader, the voting process is to determine the Leader.

During the voting process, any sentinel that wants to become the Leader needs to meet two conditions: first, it needs to obtain more than half of the affirmative votes; second, the number of votes it receives also needs to be greater than or equal to the “quorum” value configured in the sentinel’s configuration file. Taking 3 sentinels as an example, let’s assume that quorum is set to 2. In this case, any sentinel that wants to become the Leader only needs to obtain 2 affirmative votes.

You may still have some difficulty understanding this, so let me draw a picture to show the election process of 3 sentinels with a quorum of 2.

At time T1, S1 determines that the primary database is “objectively offline” and wants to become the Leader. It first votes for itself, and then sends commands to S2 and S3, indicating its intention to become the Leader.

At time T2, S3 determines that the primary database is “objectively offline” and it also wants to become the Leader. So it votes for itself first, and then sends commands to S1 and S2, indicating its intention to become the Leader.

At time T3, S1 receives the Leader voting request from S3. Since S1 has already voted for itself with a Y, it cannot vote for other sentinels anymore. Therefore, S1 replies with N, indicating disagreement. At the same time, S2 receives the Leader voting request from S3 that was sent at T2. Since S2 has not voted before, it will reply with Y to the first sentinel that sends it a voting request, and with N to subsequent voting requests. Therefore, at T3, S2 replies to S3, agreeing that S3 should become the Leader.

At time T4, S2 finally receives the voting command sent by S1 at T1. Since S2 had already agreed to S3’s voting request at T3, it replies to S1 with N, indicating disagreement with S1 becoming the Leader. This situation occurs because the network transmission between S3 and S2 is normal, while the network transmission between S1 and S2 may be congested, causing the voting request to be slow.

Finally, at time T5, S1 receives one affirmative vote Y from itself and one dissenting vote N from S2. On the other hand, S3 not only receives one affirmative vote Y from itself but also receives one affirmative vote Y from S2. At this point, S3 not only obtains more than half of the affirmative votes but also reaches the preset quorum value (quorum is 2), so it ultimately becomes the Leader. After that, S3 will start the leader election process and notify other slaves and clients of the information of the new primary database.

If S3 does not obtain 2 affirmative votes, then this round of voting will not produce a Leader. The sentinel cluster will wait for a period of time (twice the timeout for sentinel failover) and then start a new election. This is because the successful voting of the sentinel cluster depends to a large extent on the normal transmission of the election command. If there is high network pressure or temporary blockage, no sentinel may be able to obtain more than half of the affirmative votes. Therefore, waiting for the network congestion to improve before conducting a new round of voting increases the probability of success.

It should be noted that if the sentinel cluster only has 2 instances, in this case, a sentinel must obtain 2 votes instead of 1 to become the Leader. Therefore, if one sentinel goes down, the cluster will not be able to perform the master-slave switch. Therefore, it is usually recommended to configure at least 3 sentinel instances. This is very important and should not be overlooked in practical applications.

Summary #

Usually, when solving a system problem, we introduce a new mechanism or design a new layer of functionality, just like the content we learned in these two lessons: to achieve master-slave switching, we introduce sentinels; to prevent a single sentinel from failing to perform master-slave switching and to reduce misjudgment rate, we introduce sentinel clusters; sentinel clusters also need some mechanisms to support their normal operation.

In this lesson, I introduced to you the key mechanisms that support sentinel clusters, including:

• The process of sentinel cluster formation based on pub/sub mechanism;
• The list of slave nodes based on the INFO command, which helps sentinels establish connections with slave nodes;
• The pub/sub functionality based on sentinels themselves, which realizes event notification between clients and sentinels.

For master-slave switching, of course, it cannot be performed by any sentinel that wants to execute it, otherwise it will be chaotic. Therefore, the sentinel cluster needs to determine the “objectively offline” state of the master database, and after voting arbitration, elect a leader who is responsible for the actual master-slave switching. The leader is responsible for selecting the new master database and notifying the slave nodes and clients.

Finally, I would like to share an experience with you: Ensure that the configurations of all sentinel instances are consistent, especially the value for subjective offline judgment down-after-milliseconds. We have encountered a “pitfall” before. At that time, in our project, because this value was configured differently on different sentinel instances, the sentinel cluster could not reach a consensus on the faulty master database, and the master-slave switch could not be performed in a timely manner. The result was an unstable cluster service. So, you must not ignore this seemingly simple experience.

One Question for Each Lesson #

In this lesson, I will ask you a small question.

Assume there is a Redis cluster with one master and four slaves, and the cluster is also configured with five Sentinel instances. The quorum value is set to 2. During operation, if three Sentinel instances fail, can the Redis master still correctly determine that it is “objectively offline”? If so, can automatic failover between the master and the slaves still occur? Furthermore, is it better to have more Sentinel instances? And if we increase the down-after-milliseconds value, does it help to reduce misjudgments?

Feel free to discuss and exchange ideas with me in the comments section. If you have friends who are interested in learning about Sentinel clusters, please feel free to share today’s content with them and help them solve problems together. See you in the next class.