39 Practical Redis Cluster Mode Part 1

39 Practical Redis Cluster Mode Part 1 #

Redis Cluster is a Redis cluster solution introduced in Redis 3.0. It distributes data across different service areas to reduce the system’s reliance on a single master node and greatly improve the read and write performance of the Redis service.

Redis divides all data into 16,384 slots. Each node is responsible for a portion of these slots. When a Redis client connects to the cluster, it receives a copy of the cluster’s slot allocation information. With this information, the client can directly send the request commands to the corresponding nodes for processing.

Redis Cluster operates in a decentralized mode without proxies. The vast majority of commands sent by clients are directly executed by the relevant nodes. In most cases, the request commands are processed and responded to without forwarding or only forwarding once. Therefore, the performance of each node in the cluster is very similar to that of a standalone Redis server. In theory, doubling the number of master nodes through horizontal scaling would also double the performance of request processing. Therefore, Redis Cluster has very high performance.

The architecture diagram of Redis Cluster is as follows:

Building Redis Cluster #

There are two ways to build Redis Cluster: using the create-cluster tool provided in the Redis source code to quickly set up the Redis cluster environment or manually creating the Redis cluster environment using configuration files.

Quick setup of Redis Cluster #

The create-cluster tool is located in the utils/create-cluster directory, as shown in the following image:

You can quickly create a Redis cluster by running the command ./create-cluster start, as shown below:

$ ./create-cluster start # Create cluster
Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006

Next, we need to use the create command to form a cluster with the 6 nodes created above. Run the following command:

[@iZ2ze0nc5n41zomzyqtksmZ:create-cluster]$ ./create-cluster create # Build cluster
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:30005 to 127.0.0.1:30001
Adding replica 127.0.0.1:30006 to 127.0.0.1:30002
Adding replica 127.0.0.1:30004 to 127.0.0.1:30003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 445f2a86fe36d397613839d8cc1ae6702c976593 127.0.0.1:30001
   slots:[0-5460] (5461 slots) master
M: 63bb14023c0bf58926738cbf857ea304bff8eb50 127.0.0.1:30002
   slots:[5461-10922] (5462 slots) master
M: 864d4dfe32e3e0b81a64cec8b393bbd26a65cbcc 127.0.0.1:30003
   slots:[10923-16383] (5461 slots) master
S: 64828ab44566fc5ad656e831fd33de87be1387a0 127.0.0.1:30004
   replicates 445f2a86fe36d397613839d8cc1ae6702c976593
S: 0b17b00542706343583aa73149ec5ff63419f140 127.0.0.1:30005
   replicates 63bb14023c0bf58926738cbf857ea304bff8eb50
S: e35f06ca9b700073472d72001a39ea4dfcb541cd 127.0.0.1:30006
   replicates 864d4dfe32e3e0b81a64cec8b393bbd26a65cbcc
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 445f2a86fe36d397613839d8cc1ae6702c976593 127.0.0.1:30001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 864d4dfe32e3e0b81a64cec8b393bbd26a65cbcc 127.0.0.1:30003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: e35f06ca9b700073472d72001a39ea4dfcb541cd 127.0.0.1:30006
   slots: (0 slots) slave
   replicates 864d4dfe32e3e0b81a64cec8b393bbd26a65cbcc
S: 0b17b00542706343583aa73149ec5ff63419f140 127.0.0.1:30005
   slots: (0 slots) slave
   replicates 63bb14023c0bf58926738cbf857ea304bff8eb50
M: 63bb14023c0bf58926738cbf857ea304bff8eb50 127.0.0.1:30002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 64828ab44566fc5ad656e831fd33de87be1387a0 127.0.0.1:30004
   slots: (0 slots) slave
   replicates 445f2a86fe36d397613839d8cc1ae6702c976593
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

During the process, you will be asked whether to use 30001, 30002, and 30003 as master nodes and 30004, 30005, and 30006 as the respective slave nodes. After typing yes, the process will be completed.

We can connect to the cluster using redis-cli. Use the following command:

$ redis-cli -c -p 30001

To view the information of cluster nodes, use the following command:

127.0.0.1:30001> cluster nodes
864d4dfe32e3e0b81a64cec8b393bbd26a65cbcc 127.0.0.1:30003@40003 master - 0 1585125835078 3 connected 10923-16383
e35f06ca9b700073472d72001a39ea4dfcb541cd 127.0.0.1:30006@40006 slave 864d4dfe32e3e0b81a64cec8b393bbd26a65cbcc 0 1585125835078 6 connected
0b17b00542706343583aa73149ec5ff63419f140 127.0.0.1:30005@40005 slave 63bb14023c0bf58926738cbf857ea304bff8eb50 0 1585125835078 5 connected
63bb14023c0bf58926738cbf857ea304bff8eb50 127.0.0.1:30002@40002 master - 0 1585125834175 2 connected 5461-10922
445f2a86fe36d397613839d8cc1ae6702c976593 127.0.0.1:30001@40001 myself,master - 0 1585125835000 1 connected 0-5460
64828ab44566fc5ad656e831fd33de87be1387a0 127.0.0.1:30004@40004 slave 445f2a86fe36d397613839d8cc1ae6702c976593 0 1585125835000 4 connected

From the above output, we can see that 30001, 30002, and 30003 are master nodes. The slots assigned to node 30001 are 0~5460, node 30002 is assigned 5461~10922, and node 30003 is assigned 10923~16383. There are a total of 16384 slots (0~16383).

Although the create-cluster method is fast, it has fixed numbers of master and slave nodes, as well as a fixed slot allocation pattern. Moreover, all nodes are installed on the same server, so it can only be used for testing purposes.

Once our testing is complete, we can use the following command to close and clean the cluster:

$ ./create-cluster stop # Stop the cluster
Stopping 30001
Stopping 30002
Stopping 30003
Stopping 30004
Stopping 30005
Stopping 30006
$ ./create-cluster clean # Clean the cluster

Manually Building Redis Cluster #

Due to the limitations of the create-cluster tool, in a real production environment, we need to build the Redis cluster manually by adding configurations. First, we need to copy the Redis installation package to the node1 to node6 directories. In this case, we need to install 6 nodes, 3 masters and 3 slaves, as shown in the following diagram:

Next, we will configure and start the Redis cluster.

1. Configure the files

We need to modify the redis.conf file in each node to set cluster-enabled yes to enable cluster mode and change the port number using port 3000X.

2. Start the nodes

After configuring the redis.conf file, we can start all nodes using the following command:

cd /usr/local/soft/mycluster/node1
./src/redis-server redis.conf

3. Create the cluster and assign slots

Although we have started 6 nodes, they are not connected to each other and each is in its own cluster. Therefore, we need to connect these nodes to form a cluster and assign them slots. Use the following command:

redis-cli --cluster create 127.0.0.1:30001 127.0.0.1:30002 127.0.0.1:30003 127.0.0.1:30004 127.0.0.1:30005 127.0.0.1:30006 --cluster-replicas 1

In the above command, create is followed by multiple nodes to specify them as nodes of the entire cluster. cluster-replicas specifies the number of replicas for each master node, with 1 indicating one slave for each master.

After executing the create command, the system will assign roles and slot allocation plans to the nodes, as shown below:

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0-5460
Master[1] -> Slots 5461-10922
Master[2] -> Slots 10923-16383
Adding replica 127.0.0.1:30005 to 127.0.0.1:30001
Adding replica 127.0.0.1:30006 to 127.0.0.1:30002
Adding replica 127.0.0.1:30004 to 127.0.0.1:30003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: bdd1c913f87eacbdfeabc71befd0d06c913c891c 127.0.0.1:30001
   slots:[0-5460] (5461 slots) master
M: bdd1c913f87eacbdfeabc71befd0d06c913c891c 127.0.0.1:30002
   slots:[5461-10922] (5462 slots) master
M: bdd1c913f87eacbdfeabc71befd0d06c913c891c 127.0.0.1:30003
   slots:[10923-16383] (5461 slots) master
S: bdd1c913f87eacbdfeabc71befd0d06c913c891c 127.0.0.1:30004
   replicates bdd1c913f87eacbdfeabc71befd0d06c913c891c
S: bdd1c913f87eacbdfeabc71befd0d06c913c891c 127.0.0.1:30005
   replicates bdd1c913f87eacbdfeabc71befd0d06c913c891c
S: bdd1c913f87eacbdfeabc71befd0d06c913c891c 127.0.0.1:30006
   replicates bdd1c913f87eacbdfeabc71befd0d06c913c891c

   replicates bdd1c913f87eacbdfeabc71befd0d06c913c891c
Can I set the above configuration? (type 'yes' to accept):

From the above information, it can be seen that Redis plans to set 30001, 30002, and 30003 as master nodes and assign them slots. The slot range for 30001 is 0~5460, for 30002 is 5461~10922, and for 30003 is 10923~16383. It also sets 30005 as the replica of 30001, 30006 as the replica of 30002, and 30004 as the replica of 30003. We only need to enter yes to confirm and execute the assignment, as shown below:

Can I set the above configuration? (type ‘yes’ to accept): yes

Nodes configuration updated Assign a different config epoch to each node Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join …. Performing Cluster Check (using node 127.0.0.1:30001) M: 887397e6fefe8ad19ea7569e99f5eb8a803e3785 127.0.0.1:30001 slots:[0-5460] (5461 slots) master 1 additional replica(s) S: abec9f98f9c01208ba77346959bc35e8e274b6a3 127.0.0.1:30005 slots: (0 slots) slave replicates 887397e6fefe8ad19ea7569e99f5eb8a803e3785 S: 1a324d828430f61be6eaca7eb2a90728dd5049de 127.0.0.1:30004 slots: (0 slots) slave replicates f5958382af41d4e1f5b0217c1413fe19f390b55f S: dc0702625743c48c75ea935c87813c4060547cef 127.0.0.1:30006 slots: (0 slots) slave replicates 3da35c40c43b457a113b539259f17e7ed616d13d M: 3da35c40c43b457a113b539259f17e7ed616d13d 127.0.0.1:30002 slots:[5461-10922] (5462 slots) master 1 additional replica(s) M: f5958382af41d4e1f5b0217c1413fe19f390b55f 127.0.0.1:30003 slots:[10923-16383] (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. Check for open slots… Check slots coverage… [OK] All 16384 slots covered.

The ‘OK’ indicates that the entire cluster has started successfully.

Next, we will use the redis-cli to connect to and test the running status of the cluster, as shown in the code below:

$ redis-cli -c -p 30001 # Connect to the cluster 127.0.0.1:30001> cluster info # View cluster information cluster_state:ok # Normal state cluster_slots_assigned:16384 # Number of slots cluster_slots_ok:16384 # Number of normal slots cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 # Number of nodes in the cluster cluster_size:3 # Number of master nodes in the cluster cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:130 cluster_stats_messages_pong_sent:127 cluster_stats_messages_sent:257 cluster_stats_messages_ping_received:122 cluster_stats_messages_pong_received:130 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:257

The explanation of the related fields has been marked in the above code, so it won’t be repeated here.

Dynamically Add or Remove Nodes #

In some cases, we need to dynamically add or remove nodes to a running cluster based on the actual business situation. Here are the operations we need to perform.

Add Master Nodes #

Adding Method 1: cluster meet

We can use the cluster meet ip:port command to add a node to the cluster. Execute the command as follows:

127.0.0.1:30001> cluster meet 127.0.0.1 30007 OK 127.0.0.1:30001> cluster nodes dc0702625743c48c75ea935c87813c4060547cef 127.0.0.1:30006@40006 slave 3da35c40c43b457a113b539259f17e7ed616d13d 0 1585142916000 6 connected df0190853a53d8e078205d0e2fa56046f20362a7 127.0.0.1:30007@40007 master - 0 1585142917740 0 connected f5958382af41d4e1f5b0217c1413fe19f390b55f 127.0.0.1:30003@40003 master - 0 1585142916738 3 connected 10923-16383 3da35c40c43b457a113b539259f17e7ed616d13d 127.0.0.1:30002@40002 master - 0 1585142913000 2 connected 5461-10922 abec9f98f9c01208ba77346959bc35e8e274b6a3 127.0.0.1:30005@40005 slave 887397e6fefe8ad19ea7569e99f5eb8a803e3785 0 1585142917000 5 connected 887397e6fefe8ad19ea7569e99f5eb8a803e3785 127.0.0.1:30001@40001 myself,master - 0 1585142915000 1 connected 0-5460 1a324d828430f61be6eaca7eb2a90728dd5049de 127.0.0.1:30004@40004 slave f5958382af41d4e1f5b0217c1413fe19f390b55f 0 1585142916000 4 connected

It can be seen that the node with port 30007 has been added to the cluster and set as a master node.

Adding Method 2: add-node To add a node to the cluster using redis-cli --cluster add-node, execute the command as follows:

$ redis-cli --cluster add-node 127.0.0.1:30008 127.0.0.1:30001
>>> Adding node 127.0.0.1:30008 to cluster 127.0.0.1:30001
>>> Performing Cluster Check (using node 127.0.0.1:30001)
M: 887397e6fefe8ad19ea7569e99f5eb8a803e3785 127.0.0.1:30001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: dc0702625743c48c75ea935c87813c4060547cef 127.0.0.1:30006
   slots: (0 slots) slave
   replicates 3da35c40c43b457a113b539259f17e7ed616d13d
M: df0190853a53d8e078205d0e2fa56046f20362a7 127.0.0.1:30007
   slots: (0 slots) master
M: f5958382af41d4e1f5b0217c1413fe19f390b55f 127.0.0.1:30003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 1d09d26fd755298709efe60278457eaa09cefc26 127.0.0.1:30008
   slots: (0 slots) master
M: 3da35c40c43b457a113b539259f17e7ed616d13d 127.0.0.1:30002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: abec9f98f9c01208ba77346959bc35e8e274b6a3 127.0.0.1:30005
   slots: (0 slots) slave
   replicates 887397e6fefe8ad19ea7569e99f5eb8a803e3785
S: 1a324d828430f61be6eaca7eb2a90728dd5049de 127.0.0.1:30004
   slots: (0 slots) slave
   replicates f5958382af41d4e1f5b0217c1413fe19f390b55f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[ERR] Node 127.0.0.1:30008 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

From the above result, it can be seen that node 30008 has also been set as a master node.

Adding a slave node #

To set the current node as a replica node for the target node, use the command cluster replicate nodeId. Execute the command as follows:

127.0.0.1:30008> cluster replicate df0190853a53d8e078205d0e2fa56046f20362a7
OK
127.0.0.1:30008> cluster nodes

df0190853a53d8e078205d0e2fa56046f20362a7 127.0.0.1:30007@40007 master - 0 1585147827000 0 connected
abec9f98f9c01208ba77346959bc35e8e274b6a3 127.0.0.1:30005@40005 slave 887397e6fefe8ad19ea7569e99f5eb8a803e3785 0 1585147827000 1 connected
1a324d828430f61be6eaca7eb2a90728dd5049de 127.0.0.1:30004@40004 slave f5958382af41d4e1f5b0217c1413fe19f390b55f 0 1585147823000 3 connected
887397e6fefe8ad19ea7569e99f5eb8a803e3785 127.0.0.1:30001@40001 master - 0 1585147826000 1 connected 0-5460
dc0702625743c48c75ea935c87813c4060547cef 127.0.0.1:30006@40006 slave 3da35c40c43b457a113b539259f17e7ed616d13d 0 1585147826930 2 connected
f5958382af41d4e1f5b0217c1413fe19f390b55f 127.0.0.1:30003@40003 master - 0 1585147826000 3 connected 10923-16383
1d09d26fd755298709efe60278457eaa09cefc26 127.0.0.1:30008@40008 myself,slave df0190853a53d8e078205d0e2fa56046f20362a7 0 1585147823000 7 connected
3da35c40c43b457a113b539259f17e7ed616d13d 127.0.0.1:30002@40002 master - 0 1585147827933 2 connected 5461-10922

You can see that 30008 has become a slave node of 30007.

Deleting Nodes #

Use the cluster forget nodeId command to remove a node from the cluster.

Unlike the meet command, which only requires the IP and port, the cluster forget command requires the node ID to be specified. You can use the cluster nodes command to view the ID of all the nodes in the cluster. The first 40 characters of each line represent the node ID, as shown in the following image:

You can execute the command as follows:

127.0.0.1:30001> cluster forget df0190853a53d8e078205d0e2fa56046f20362a7
OK

Now, let’s use the cluster nodes command to view all the nodes in the cluster:

127.0.0.1:30001> cluster nodes
dc0702625743c48c75ea935c87813c4060547cef 127.0.0.1:30006@40006 slave 3da35c40c43b457a113b539259f17e7ed616d13d 0 1585143789940 6 connected
f5958382af41d4e1f5b0217c1413fe19f390b55f 127.0.0.1:30003@40003 master - 0 1585143791000 3 connected 10923-16383
3da35c40c43b457a113b539259f17e7ed616d13d 127.0.0.1:30002@40002 master - 0 1585143789000 2 connected 5461-10922
abec9f98f9c01208ba77346959bc35e8e274b6a3 127.0.0.1:30005@40005 slave 887397e6fefe8ad19ea7569e99f5eb8a803e3785 0 1585143789000 5 connected
887397e6fefe8ad19ea7569e99f5eb8a803e3785 127.0.0.1:30001@40001 myself,master - 0 1585143786000 1 connected 0-5460
1a324d828430f61be6eaca7eb2a90728dd5049de 127.0.0.1:30004@40004 slave f5958382af41d4e1f5b0217c1413fe19f390b55f 0 1585143791945 4 connected

As you can see, the node on port 30007 has been successfully removed.

Summary #

This article explained the two ways of creating a Redis cluster: using create-cluster start and cluster create. While the first method is faster, it can only create a fixed number of master and slave nodes and requires all nodes to be on the same server, making it suitable only for testing environments. We also learned how to dynamically add and delete nodes in a Redis cluster.