40 Practical Redis Cluster Mode Part 2

40 Practical Redis Cluster Mode Part 2 #

In the previous article, we discussed the setup of Redis cluster and the dynamic addition and removal of nodes. Let’s briefly review. The cluster we initially built consisted of nodes 30001 to 30006. Nodes 30007 and 30008 were later added as master and slave nodes, respectively. We can use the --cluster info command to see the distribution of master nodes and slots. The executed code is as follows:

$ redis-cli --cluster info 127.0.0.1:30001
127.0.0.1:30001 (887397e6...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30007 (df019085...) -> 0 keys | 0 slots | 1 slaves.
127.0.0.1:30003 (f5958382...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:30002 (3da35c40...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 4 masters.
0.00 keys per slot on average.

As we can see, the dynamically added master node 30007 has one slave node, but no slots are assigned to it. This obviously cannot meet our requirements because we only added nodes without processing any data. Therefore, we need to re-shard and distribute data across all master nodes in order to fully utilize the cluster.

Resharding #

We can use the redis-cli --cluster reshard command to reassign slots. Execute the command as follows:

$ redis-cli --cluster reshard 127.0.0.1:30007
>>> Performing Cluster Check (using node 127.0.0.1:30007)
M: df0190853a53d8e078205d0e2fa56046f20362a7 127.0.0.1:30007
   slots:[0-1332],[5461-6794],[10923-12255] (4000 slots) master
   1 additional replica(s)
S: dc0702625743c48c75ea935c87813c4060547cef 127.0.0.1:30006
   slots: (0 slots) slave
   replicates 3da35c40c43b457a113b539259f17e7ed616d13d
M: 3da35c40c43b457a113b539259f17e7ed616d13d 127.0.0.1:30002
   slots:[6795-10922] (4128 slots) master
   1 additional replica(s)
S: 1a324d828430f61be6eaca7eb2a90728dd5049de 127.0.0.1:30004
   slots: (0 slots) slave
   replicates f5958382af41d4e1f5b0217c1413fe19f390b55f
S: 1d09d26fd755298709efe60278457eaa09cefc26 127.0.0.1:30008
   slots: (0 slots) slave
   replicates df0190853a53d8e078205d0e2fa56046f20362a7
S: abec9f98f9c01208ba77346959bc35e8e274b6a3 127.0.0.1:30005
   slots: (0 slots) slave
   replicates 887397e6fefe8ad19ea7569e99f5eb8a803e3785
M: f5958382af41d4e1f5b0217c1413fe19f390b55f 127.0.0.1:30003
   slots:[12256-16383] (4128 slots) master
   1 additional replica(s)
M: 887397e6fefe8ad19ea7569e99f5eb8a803e3785 127.0.0.1:30001
   slots:[1333-5460] (4128 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?

During the execution, it will ask you how many slots you want to move, with a range of 1 to 16384. Here, we input 4000, which means moving 4000 slots to a specific master node. After entering the command, the executed effect is as follows:

How many slots do you want to move (from 1 to 16384)? 4000
What is the receiving node ID?

Then it will ask you to which node you want to allocate these slots. Please enter the node ID of port 30007 mentioned above, and press Enter. The executed effect is as follows:

How many slots do you want to move (from 1 to 16384)? 4000
What is the receiving node ID? df0190853a53d8e078205d0e2fa56046f20362a7

Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you have entered all the source node IDs.
Source node #1:

At this point, it will ask you from which source node to perform the transfer. Enter the all command to randomly select from all nodes. The execution result is as follows:

# ...... Ignore other lines
Moving slot 2656 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2657 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2658 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2659 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2660 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2661 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2662 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2663 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2664 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Moving slot 2665 from 887397e6fefe8ad19ea7569e99f5eb8a803e3785
Do you want to proceed with the proposed reshard plan (yes/no)?

At this point, it will list all the nodes to be transferred and ask for your confirmation. You just need to enter yes to start the transfer operation.

After the transfer is complete, we use the cluster slots command to view the information of the hash slots. The result is as follows:

$ redis-cli -c -p 30001
127.0.0.1:30001> cluster slots # View cluster slot information
1) 1) (integer) 0
   2) (integer) 1332
   3) 1) "127.0.0.1"
      2) (integer) 30007
      3) "df0190853a53d8e078205d0e2fa56046f20362a7"
   4) 1) "127.0.0.1"
      2) (integer) 30008
      3) "1d09d26fd755298709efe60278457eaa09cefc26"
2) 1) (integer) 5461
   2) (integer) 6794
   3) 1) "127.0.0.1"
      2) (integer) 30007
      3) "df0190853a53d8e078205d0e2fa56046f20362a7"
   4) 1) "127.0.0.1"
      2) (integer) 30008
      3) "1d09d26fd755298709efe60278457eaa09cefc26"
3) 1) (integer) 10923
   2) (integer) 12255
   3) 1) "127.0.0.1"
      2) (integer) 30007
      3) "df0190853a53d8e078205d0e2fa56046f20362a7"
   4) 1) "127.0.0.1"
      2) (integer) 30008
      3) "1d09d26fd755298709efe60278457eaa09cefc26"
4) 1) (integer) 12256
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 30003
      3) "f5958382af41d4e1f5b0217c1413fe19f390b55f"
   4) 1) "127.0.0.1"
      2) (integer) 30004
      3) "1a324d828430f61be6eaca7eb2a90728dd5049de"
5) 1) (integer) 6795
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 30002
      3) "3da35c40c43b457a113b539259f17e7ed616d13d"
    1. “127.0.0.1”
    2. (integer) 30006
    3. “dc0702625743c48c75ea935c87813c4060547cef”
    1. (integer) 1333
    2. (integer) 5460
      1. “127.0.0.1”
      2. (integer) 30001
      3. “887397e6fefe8ad19ea7569e99f5eb8a803e3785”
      1. “127.0.0.1”
      2. (integer) 30005
      3. “abec9f98f9c01208ba77346959bc35e8e274b6a3”

From the results, it can be seen that 30007 has extracted some slots from the other three master nodes as its own slots.

Note: If the error “/usr/bin/env: ruby: No such file or directory” occurs during the execution of this process, it indicates that the tool requires a Ruby environment to execute. You can use the command “yum install ruby” to install the Ruby environment.

Slot Locating Algorithm #

Redis cluster has a total of 16384 slots. Each master node is responsible for maintaining a portion of the slots and the key-value data mapped by those slots. By default, Redis cluster uses the CRC16 algorithm to hash the key value to obtain an integer value, and then takes the modulus of 16384 to determine the specific slot. The formula is as follows:

slot = CRC16(key) % 16383

Load Balancing #

In the case of unbalanced load in a Redis cluster, we can use the rebalance command to redistribute the number of slots each node is responsible for, so as to balance the load pressure on each node and improve the overall efficiency of the Redis cluster.

The rebalance command is as follows:

$ redis-cli --cluster rebalance 127.0.0.1:30007

It is important to note that even if the rebalance command is entered, it may not execute. If it determines that there is no need for allocation, it will exit directly, as shown below:

$ redis-cli --cluster rebalance 127.0.0.1:30007
>>> Performing Cluster Check (using node 127.0.0.1:30007)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
*** No rebalancing needed! All nodes are within the 2.00% threshold.

Code Example #

In the previous sections, we discussed the related functions of setting up a Redis cluster. Next, let’s use Java code to operate on the Redis cluster. In this article, we will continue to use Jedis as the client for the relevant operations. The core code is as follows:

import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import java.util.HashSet;
import java.util.Set;

public class ClusterExample {
    public static void main(String[] args) {
        // Cluster nodes information
        Set<HostAndPort> nodes = new HashSet<>();
        nodes.add(new HostAndPort("127.0.0.1", 30001));
        nodes.add(new HostAndPort("127.0.0.1", 30002));
        nodes.add(new HostAndPort("127.0.0.1", 30003));
        nodes.add(new HostAndPort("127.0.0.1", 30004));
        nodes.add(new HostAndPort("127.0.0.1", 30005));
        nodes.add(new HostAndPort("127.0.0.1", 30006));
        // Create cluster connection

JedisCluster jedisCluster = new JedisCluster(nodes,
                    10000,  // Timeout time
                    10);    // Maximum number of reconnection attempts
// Add data
String setResult = jedisCluster.set("lang", "redis");
// Output result
System.out.println("Add: " + setResult);
// Query result
String getResult = jedisCluster.get("lang");
// Output result
System.out.println("Query: " + getResult);

The execution results of the above program are as follows:

Add: OK
Query: redis

This result indicates that the Redis cluster operation is normal. Apart from the different operation objects, the method names of the operations are the same, which is more friendly for programmers. You can write the corresponding code based on your own business scenario.

Failover #

In the last part of the article, let’s take a look at some knowledge points related to Redis cluster failures. This will help us handle failure issues without panic and provide some assistance when dealing with failures.

Failure Detection #

There are two important concepts in failure detection: Possibly Fail (PFAIL) and Fail.

The health monitoring of the cluster is confirmed by regularly sending PING messages to other nodes in the cluster. If a node that sends a PING message does not receive a returned PONG message within a specified time, the other node will be marked as Possibly Fail.

When a node finds that another node is Possibly Fail, it broadcasts this information to the entire cluster. The other nodes receive this message and monitor whether the specific node is really offline by using PING. If a node receives the Possibly Fail message of a node more than half of the cluster, it can mark the specific node as Fail and broadcast it to the entire cluster. This forces other nodes to acknowledge the fact that the node has already failed and immediately perform master-slave switch for the disconnected node.

This is similar to the concepts of subjective and objective failures in the sentinel mode.

Failover #

After a node is identified by the cluster as Fail, failover can be performed. The execution process of failover is as follows:

  1. Select a slave node from all the slave nodes of the failed master node (the selection method is detailed in the “Principles for Electing a New Master Node” section below);
  2. The slave node executes the SLAVEOF NO ONE command, disables the replication function of this slave node, and becomes a master node from a slave node. The previously synchronized dataset will not be discarded;
  3. The new master node revokes all slot assignments to the failed master node and assigns all these slots to itself;
  4. The new master node broadcasts a PONG message to the cluster. This PONG message lets other nodes in the cluster know that this node has become the master node from a slave node and has taken over the slot information originally handled by the failed node;
  5. The new master node starts processing related command requests, completing the failover process.

Principles for Electing a New Master Node #

The method for electing a new master node is as follows:

  1. The epoch of the cluster is a counter incremented by itself, and its initial value is 0;
  2. Each master node has one opportunity to vote, and the master node will vote for the first requesting slave node;
  3. When a slave node finds that the master node it is replicating is Fail, it broadcasts a message to the cluster, requesting all master nodes with voting rights to vote for this slave node;
  4. If a master node with voting rights has not voted for anyone else, it will send a message to the first requesting slave node, indicating that it will vote for this slave node;
  5. When the slave node receives votes greater than more than half of the cluster, it will be elected as the new master node.

At this point, the selection of the new master node is completed.

Summary #

This article starts with dynamically adding a master node and redistributing slot using the reshard command, explains the algorithm for slot positioning and the implementation method for load balancing. It also demonstrates how to operate Redis cluster in a program using code. Finally, it explains the entire process of failure detection, failover, and electing a new master node in Redis cluster. I hope this helps you understand Redis cluster.