22 Does Sentinel Initialize the Same as Redis Instances

22 Does Sentinel Initialize the Same as Redis Instances #

In this lesson, let’s take a look at how Redis implements the sentinel mechanism in its source code.

As we know, Redis replication is an important means to ensure the availability of Redis. Once the Redis master node fails, the sentinel mechanism triggers a failover. The process of failover is actually quite complicated and involves key operations such as sentinel leader election, new master node election, and failover. However, this failover process is a common development requirement when implementing high availability systems.

So starting from this lesson, I will introduce to you one by one the sentinel mechanism of Redis and the key technical design and implementation of failover. Through learning this part, you will not only understand how the Raft protocol, which plays an important role in the failover process, is implemented, but also grasp how the notification of switch between the master node, slave node, and client is completed during failover.

However, before we start understanding the key technologies of failover, today we will first understand the initialization and basic operation process of the sentinel instance itself. This is because from the perspective of source code, the implementation of sentinel instances and regular Redis instances are both part of the same codebase, and they share some execution processes. So understanding this part of the content can also help us have a clearer understanding of the implementation mechanism of sentinel instances.

Great, now let’s take a look at the initialization process of the sentinel instance below.

Initialization of Sentinel Instance #

Since the sentinel instance is a Redis server running in a special mode, and I have already introduced the overall execution process of the main entry function after Redis server startup in [Lesson 8]. In fact, this process includes the initialization operation of the sentinel instance.

Therefore, the initialization entry function of the sentinel instance is also main (in the server.c file). When the main function is running, it will judge whether the current running logic corresponds to the sentinel instance by checking the running parameters. Specifically, before calling the initServerConfig function to initialize various configuration options, the main function will call the checkForSentinelMode function to determine if the current running instance is a sentinel instance. The code is as follows:

server.sentinel_mode = checkForSentinelMode(argc,argv);

The checkForSentinelMode function (in the server.c file) takes two parameters, the startup command string argv and the number of arguments in the startup command argc received by the main function. Then, it will judge whether a sentinel instance is running based on the following two conditions.

  • Condition 1: Whether the executed command itself, which is argv[0], is “redis-sentinel”.
  • Condition 2: Whether the executed command parameters include “–sentinel”.

The code for this part is as follows:

int checkForSentinelMode(int argc, char **argv) {
    int j;
    //The first condition checks whether the executed command itself is redis-sentinel
    if (strstr(argv[0],"redis-sentinel") != NULL) return 1;
    for (j = 1; j < argc; j++)
        //The second condition checks whether the command parameters include "--sentinel"
        if (!strcmp(argv[j],"--sentinel")) return 1;
    return 0;
}

In fact, these two conditions correspond to two ways of starting the sentinel instance from the command line. One is to directly run the redis-sentinel command, and the other is to run the redis-server command with the “–sentinel” parameter, as shown below:

redis-sentinel sentinel.conf file path
or
redis-server sentinel.conf file path --sentinel

Therefore, if either of these two conditions is met, the member variable sentinel_mode of the global variable server will be set to 1, indicating that the currently running instance is a sentinel instance. In this way, the configuration item server.sentinel_mode will be used in other parts of the source code to determine whether the currently running instance is a sentinel instance.

Initialization of Configuration Items #

After the judgment of the sentinel instance running is completed, the main function will then call the initServerConfig function to initialize various configuration items. However, because the configuration items used by the sentinel instance during runtime are different from those of the Redis instance, the main function will specifically call the initSentinelConfig and initSentinel functions to complete the initialization of the sentinel instance-specific configuration items, as shown below:

if (server.sentinel_mode) {
   initSentinelConfig();
   initSentinel();
}

The initSentinelConfig and initSentinel functions are implemented in the sentinel.c file.

Among them, the initSentinelConfig function mainly changes the port number of the current server to the dedicated port number for the sentinel instance, REDIS_SENTINEL_PORT. This is a macro definition with a default value of 26379. In addition, this function sets the server’s protected_mode to 0, which allows external connections to the sentinel instance instead of only being able to connect to the server through 127.0.0.1.

The initSentinel function further completes the initialization of the sentinel instance based on the initSentinelConfig function. This includes two main tasks.

  • First, the initSentinel function will replace the command table that the server can execute.

When the initServerConfig function is executed, the Redis server will initialize a command table and save it in the commands member variable of the global variable server. This command table itself is a hash table, where each hash item’s key corresponds to a command name and the value corresponds to the actual implementation function of that command.

Because the sentinel instance runs on a special mode of the Redis server, the commands it executes are different from the Redis instance. Therefore, the initSentinel function will clear the command table corresponding to server.commands, and then add the sentinel commands to it, as shown below:

dictEmpty(server.commands, NULL);
for (j = 0; j < sizeof(sentinelcmds) / sizeof(sentinelcmds[0]); j++) {
    ...
    struct redisCommand *cmd = sentinelcmds + j;
    retval = dictAdd(server.commands, sdsnew(cmd->name), cmd);
    ...
}

From this code, you can see that the commands that the sentinel instance can execute are saved in the sentinelcmds array, which is defined in the sentinel.c file.

Note that although some of the command names executed by the sentinel instance are the same as those in the Redis instance command table, their implementation functions are specifically implemented for the sentinel instance. For example, both the sentinel instance and the Redis instance can execute the publish, info, and role commands, but in the sentinel instance, these three commands are implemented by the functions sentinelPublishCommand, sentinelInfoCommand, and sentinelRoleCommand defined in the sentinel.c file. So, when you need to understand the implementation of commands executed by the sentinel instance in detail, be careful not to look in the wrong code file.

The following code also shows some of the commands in the command table of the sentinel instance. You can take a look:

struct redisCommand sentinelcmds[] = {
    {"ping", pingCommand, 1, "", 0, NULL, 0, 0, 0, 0, 0},
    {"sentinel", sentinelCommand, -2, "", 0, NULL, 0, 0, 0, 0, 0},
    ...
    {"publish", sentinelPublishCommand, 3, "", 0, NULL, 0, 0, 0, 0, 0},
    {"info", sentinelInfoCommand, -1, "", 0, NULL, 0, 0, 0, 0, 0},
    {"role", sentinelRoleCommand, 1, "l", 0, NULL, 0, 0, 0, 0, 0},
    ...
};
  • Second, after replacing the command table, the initSentinel function will then start to initialize various properties used by the sentinel instance.

To store these properties, the sentinel instance defines the sentinelState structure (in the sentinel.c file), which includes the ID of the sentinel instance, the current epoch for failover, the monitored master nodes, the number of currently executing scripts, and the IP and port numbers sent to other sentinel instances, among other information. The following code shows some of the properties defined in the sentinelState structure. You can take a look:

struct sentinelState {
    char myid[CONFIG_RUN_ID_SIZE + 1]; // ID of the sentinel instance
    uint64_t current_epoch;            // Current epoch
    dict *masters;                     // Hash table of monitored master nodes
    int tilt;                          // Whether in TILT mode
    int running_scripts;               // Number of running scripts
    mstime_t tilt_start_time;          // TILT mode start time
    mstime_t previous_time;            // Previous time handler execution time
    list *scripts_queue;               // Queue for storing scripts
    char *announce_ip;                 // IP information sent to other sentinel instances
    int announce_port;                 // Port number sent to other sentinel instances
    ...
} sentinel;

This way, the initSentinel function will mainly set these attributes to their initial values. For example, it will create a hash table for the monitored master nodes, where the key of each hash item records the name of the master node, and the value records the corresponding data structure pointer.

So far, the initialization of the configuration options for the sentinel instance is completed. The following diagram shows this initialization process, which you can review.

Next, the main function will call the initServer function to complete the initialization of the server itself, which will also be executed by the sentinel instance. Then, the main function will call the sentinelIsRunning function (in the sentinel.c file) to start the sentinel instance.

Starting the Sentinel Instance #

The sentinelIsRunning function has a simple execution logic. It first confirms that the configuration file of the sentinel instance exists and can be written to properly. Then, it checks if the sentinel instance has an ID set. If no ID is set, the sentinelIsRunning function will randomly generate an ID for the sentinel instance.

Finally, the sentinelIsRunning function calls the sentinelGenerateInitialMonitorEvents function (in the sentinel.c file) to send event information to each monitored master node. The following diagram shows the basic execution flow of the sentinelIsRunning function, which you can take a look at.

So, how does the sentinelIsRunning function obtain the address information of the master nodes?

This is related to the initSentinel function I just mentioned, which initializes the data structure sentinel.masters of the sentinel instance. This structure uses a hash table to record the monitored master nodes, and each master node is saved using the sentinelRedisInstance structure. In the sentinelRedisInstance structure, the address information of the monitored master node is included. This address information is saved by the sentinelAddr structure, which includes the IP and port number of the node, as shown below:

typedef struct sentinelAddr {
    char *ip;
    int port;
} sentinelAddr;

In addition, the sentinelRedisInstance structure also includes some other information related to the master node and failover, such as the master node’s name, ID, other sentinel instances monitoring the same master node, the slave nodes of the master node, subjective and objective downtime durations of the master node, etc. The following code shows part of the content of the sentinelRedisInstance structure, which you can take a look at:

typedef struct sentinelRedisInstance {
    int flags;      // Flags indicating the instance type and status
    char *name;     // Name of the instance
    char *runid;    // ID of the instance
    uint64_t config_epoch;  // Configuration epoch
    sentinelAddr *addr; // Address information of the instance
    ...
    mstime_t s_down_since_time; // Downtime duration in terms of subjectivity
    mstime_t o_down_since_time; // Downtime duration in terms of objectivity
    ...
    dict *sentinels;    // Other sentinel instances monitoring the same master node
   dict *slaves;   // Slave nodes of the master node
   ...
}

Here, you need to pay attention that the sentinelRedisInstance structure is a generic structure that can represent not only master nodes but also slave nodes or other sentinel instances.

The structure has a member variable flags, which can be set to different values to represent different types of instances. For example, when flags is set to the SRI_MASTER, SRI_SLAVE, or SRI_SENTINEL macro definition (in the sentinel.c file), they respectively represent the current instance as a master node, slave node, or other sentinel. When you read the source code related to the sentinel, you can see that the code will check the flags, obtain the current instance type, and then execute the corresponding code logic.

Okay, now you know that when the sentinel needs to communicate with a monitored master node, it only needs to retrieve the sentinelRedisInstance instance corresponding to the master node from the sentinel.masters structure, and then it can send messages to the master node.

You can refer to the following code for the execution logic of the sentinelGenerateInitialMonitorEvents function:

void sentinelGenerateInitialMonitorEvents(void) {
    ...

dictIterator *di;
dictEntry *de;

di = dictGetIterator(sentinel.masters); // Get the iterator for 'masters'
while ((de = dictNext(di)) != NULL) { // Get the monitored master nodes
    sentinelRedisInstance *ri = dictGetVal(de);
    sentinelEvent(LL_WARNING, "+monitor", ri, "%@ quorum %d", ri->quorum); // Send the '+monitor' event
}
dictReleaseIterator(di);
}

From the code, you can see that the sentinelGenerateInitialMonitorEvents function calls the sentinelEvent function (defined in sentinel.c) to actually send the event information.

The prototype definition of the sentinelEvent function is as follows, where the level parameter represents the current log level, type represents the subscription channel used for sending event information, ri represents the corresponding interacting master node, and fmt represents the message content to be sent.

void sentinelEvent(int level, char *type, sentinelRedisInstance *ri, const char *fmt, ...)

Now, the sentinelEvent function first checks whether the first two characters of the passed message content are “%” and “@”. If so, it checks whether the monitored instance is a master node. If it is, the sentinelEvent function adds the name, IP, and port number of the monitored instance to the message to be sent, as shown below:

… // If the message starts with “%” and “@”, check if the instance is a master node if (fmt[0] == ‘%’ && fmt[1] == ‘@’) { // Check if the instance’s flags are SRI_MASTER, indicating that it is a master node sentinelRedisInstance *master = (ri->flags & SRI_MASTER) ? NULL : ri->master; // If the current instance is a master node, use snprintf to generate the message ‘msg’ based on the instance’s name, IP address, port number, and other information if (master) { snprintf(msg, sizeof(msg), “%s %s %s %d @ %s %s %d”, sentinelRedisInstanceTypeStr(ri), ri->name, ri->addr->ip, ri->addr->port, master->name, master->addr->ip, master->addr->port); } … } …

Finally, the sentinelEvent function calls the pubsubPublishMessage function (defined in pubsub.c) to send the message to the corresponding channel, as shown below:

if (level != LL_DEBUG) { channel = createStringObject(type, strlen(type)); payload = createStringObject(msg, strlen(msg)); pubsubPublishMessage(channel, payload); … }

Additionally, I would like to point out that the sentinelGenerateInitialMonitorEvents function passes the parameter type as “+monitor” to the sentinelEvent function, indicating that it will send the event information to the “+monitor” channel.

The following diagram illustrates the execution flow of the sentinelEvent function for reference:

With this, the initialization of the sentinel instance is mostly complete. Next, the sentinel will communicate with the master node, listening for changes in its state. I will explain the communication process between them in more detail in the upcoming lessons.

Summary #

In today’s lesson, I introduced you to the initialization process of a sentinel instance. The sentinel instance and Redis instance share the same entry point main function. However, the sentinel instance differs from the Redis instance in terms of the runtime configuration, runtime information, supported commands, event handling, etc.

Therefore, the main function first uses the checkForSentinelMode function to determine whether the current run is for a sentinel instance, and sets the global configuration item server.sentinel_mode accordingly. This configuration item is used elsewhere in the source code to identify whether the sentinel instance is running.

When starting a sentinel instance, the main function calls the initSentinelConfig and initSentinel functions to complete the initialization of the sentinel instance. Then, the main function calls the sentinelIsRunning function to send event information to the monitored master node, thus starting the monitoring of the master node.

Finally, I would like to remind you again that from today’s lesson, we can see that after the sentinel instance is running, it starts using the Pub/Sub pattern for communication. This communication method is usually suitable for scenarios with multiple participants.

Because the sentinel instance needs to communicate not only with the master node but also with other sentinel instances and clients, using the Pub/Sub communication method can efficiently facilitate these communication processes. In the upcoming lessons, I will further introduce the use of Pub/Sub communication method in the sentinel runtime process. I hope that after completing this part of the course, you will be able to understand the implementation of this communication method.

One question per lesson #

The sentinel instance itself has a configuration file called sentinel.conf. Can you find the function that parses this configuration file during the initialization process of the sentinel instance?