01 How Redis Executes #

In previous interviews, when asked how Redis works, the answer often received is that the client sends a command to the server, and the server executes it and returns the result to the client. However, the execution details are often avoided. When further asked how the server executes the command, very few people can provide an answer. This is somewhat regrettable. We use Redis every day, but very few know the principles behind it.

For any technology, if you only stay at the stage of “knowing how to use it”, it is difficult to achieve great things and there is even a risk of being laid off or unable to find a job. I believe that if you are able to read this article, you must be someone who is actively striving for progress and wants to make a difference. So, let’s take this opportunity to delve into the execution details of Redis.

Command Execution Process #

The execution process of a command has many details, but it can generally be divided into the following steps: the client converts the command input by the user into the Redis communication protocol, then sends the content to the server through a socket connection. After receiving the content, the server converts it into a specific command to be executed, then verifies the user’s authorization information and other relevant information. After passing the authentication, the server executes the final command. Once the command is executed, relevant information recording and data statistics are performed, and then the execution result is sent back to the client. This completes the execution process of a command. In cluster mode, the master node will also synchronize the command to the slave nodes. Now, let’s take a more specific look at the execution process.

Step 1: User inputs a command

Step 2: The client converts the command to the Redis protocol and sends it to the server through a socket connection

The client and the server communicate based on socket. When the server is initialized, it creates a socket for monitoring client sockets. The source code is as follows:

void initServer(void) {
    //...
    // Enable socket event monitoring
    if (server.port != 0 &&
        listenToPort(server.port,server.ipfd,&server.ipfd_count) == C_ERR)
        exit(1);
    //...
}

Socket knowledge: After a socket is created, two buffers, the input buffer and the output buffer, are allocated. Writing data to the buffers doesn’t immediately transmit the data over the network. Instead, the data is written to the buffers, and then the TCP protocol sends the data from the buffers to the target machine. Once the data is written to the buffer, the write function can successfully return, regardless of whether the data has arrived at the target machine or when it will be sent to the network. These are responsibilities of the TCP protocol. Note: The data may be sent to the network immediately after it is written to the buffer, or it may accumulate in the buffer and a batch of multiple write operations might be sent to the network at once. This depends on factors such as the network conditions at the time and whether the current thread is idle. These factors are not controlled by the programmer. The same is true for reading functions. They also read data from the input buffer, not directly from the network.

After the socket connection is successfully established, the client converts the command into the Redis communication protocol (RESP protocol, Redis Serialization Protocol) and sends it to the server. This communication protocol is designed to ensure that the server can quickly understand the meaning of the command. Without this communication protocol, the Redis server would have to traverse all spaces to determine the meaning of the command, which would increase the computational load on the server. By directly sending the communication protocol, the client effectively delegates the parsing work to each individual client, greatly improving the speed of Redis. For example, when we enter the set key value command, the client converts this command into the protocol *3\r\n$3\r\nSET\r\n$3\r\nKEY\r\n$5\r\nVALUE\r\n and sends it to the server. For more communication protocols, please refer to the official documentation: https://redis.io/topics/protocol

Additional Knowledge: I/O Multiplexing

Redis uses I/O multiplexing to listen to multiple socket connections, which allows one thread to handle multiple requests and reduces the overhead of thread context switching. It also avoids I/O blocking operations, greatly improving the efficiency of Redis.

The mechanism of I/O multiplexing is shown in the following diagram:

In summary, the execution process of this step is as follows:

• Establish a connection with the server using sockets and I/O multiplexing techniques.
• Convert the command to the Redis communication protocol and send it to the server via the buffer.

Step 3: The server receives the command The server will first retrieve data from the input buffer and then check if the size of the data exceeds the system’s configured limit (default is 1GB). If it exceeds this limit, an error message will be returned and the client connection will be closed. The default size is shown in the following image:

After the data size validation, the server will analyze the request commands in the input buffer, extract the command parameters contained in the command request, and store them in the attributes of the client object (the server will create a Client object for each connection).

Step 4: Preparation before execution

1. Check if it is an exit command. If it is, return directly.
2. Non-null check: Check if the client object is null. If it is, return an error message.
3. Get the execution command by querying the redisCommand structure based on the attribute information stored in the client object.
4. User authentication: Clients that have not been authenticated can only execute the AUTH (authorization) command. Clients that have not been authenticated and execute commands other than AUTH will return an error message.
5. Cluster-related operations: If it is in cluster mode, redirect the command to the target node. If it is a master node, no redirection is needed.
6. Check the maximum memory limit of the server. If the server has the maximum memory limit enabled, it will first check the memory size. If the memory exceeds the maximum value, it will perform memory reclamation.
7. Persistence check: Check if the server has enabled persistence and the configuration to stop writing on persistence failure. If this configuration is enabled and there is a persistence failure, writing commands will be prohibited.
8. Minimum slave validation in cluster mode: If it is in cluster mode and the repl_min_slaves_to_write (minimum number of slaves to write) configuration is set, writing commands will be prohibited if the number of slave nodes is less than the configured value.
9. Read-only slave validation: When the server is a read-only slave, it only accepts write commands from the master node.
10. Client subscription check: When the client is subscribing to a channel, only certain commands will be executed (SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE), and other commands will be rejected.
11. Slave status validation: When the server is a slave and is not connected to a master, only status query-related commands, such as info, will be executed.
12. Server initialization validation: When the server is starting up, only commands with the loading flag will be executed, and other commands will be rejected.
13. Lua script blocking validation: When the server is blocked due to executing a Lua script, only certain commands will be executed.
14. Transaction command validation: If a transaction command is executed, the transaction will be opened and the command will be put into the waiting queue.
15. Monitor check: If the server has enabled the monitor function, it will send the executed command and related parameters to the monitor (used to monitor the running status of the server).

After these operations, the server can execute the actual command.

Step 5: Executing the final command by calling the proc function in redisCommand.

Step 6: Record and statistics after execution

1. Check if slow query logging is enabled. If enabled, slow query logs will be recorded.
2. Check if statistics information is enabled. If enabled, some statistics information will be recorded, such as the time consumed and the increment of the counter (calls) for each executed command.
3. Check if persistence is enabled. If enabled, persistence information will be recorded.
4. If there are other slave servers replicating the current server, the just executed command will be propagated to other slave servers.

Step 7: Returning the result to the client After the command is executed, the server will send the execution result to the client via the socket, and the client will display the result to the user. This completes the execution of a command.

Summary #

When a user inputs a command, the client converts the data into the Redis protocol and sends it to the server through a socket. After receiving the data, the server converts the protocol into the actual execution command. After various validations to ensure the command can be executed correctly and safely, the server calls the specific method to execute the command. After execution, relevant statistics and records are performed, and then the execution result is returned to the client. The entire execution flow is shown in the following image:

For more detailed execution process, you can refer to the Redis source code file server.c.