11 What Events Are There in Redis Event-Driven Framework #
In Lesson 9, I introduced three IO multiplexing mechanisms provided by Linux: select, poll, and epoll. These are the operating system layer support technologies for implementing the Redis event-driven framework.
In the previous lesson, I explained the basic working mechanism of the Redis event-driven framework. I introduced the Reactor model as the foundation for the event-driven framework and used the example of client connection events in IO events to explain the basic flow of the framework: from registering listening events by calling the aeCreateFileEvent
function during server initialization, to calling the aeMain
function after server initialization is complete, and then continuously executing the aeProcessEvent
function in a loop to capture and process events triggered by client requests.
However, in the previous lesson, we mainly focused on the basic flow of the framework. So at this point, you may still have some questions, such as:
- In addition to client connections, are there any other IO events that the Redis event-driven framework listens to? Does the framework also listen to other types of events besides IO events?
- What specific operations in the Redis source code correspond to the creation and handling of these events?
In today’s lesson, I will introduce the two main categories of events in the Redis event-driven framework: IO events and time events, as well as their corresponding processing mechanisms.
In fact, understanding and learning this part of the content can help us have a more comprehensive grasp of how the Redis event-driven framework processes various requests and tasks faced during the server’s runtime as events. For example, what events are used for normal client read and write requests, which functions handle them, and how is the background snapshot task started on time?
Because the event-driven framework is the core loop process of the Redis server after it starts running, understanding when to use which function to handle which type of event can be very helpful for us to troubleshoot issues encountered during the server’s runtime.
On the other hand, we can also learn how to handle both IO events and time events in a single framework. When developing server-side programs, we often need to handle periodic tasks, and the Redis implementation of handling these two types of events provides us with a good reference.
Alright, in order to have a relatively comprehensive understanding of these two types of events, let’s start by learning about the data structures and their initialization in the event-driven framework’s loop process, as they include the definition and initialization operations for these two types of events.
aeEventLoop Structure and Initialization #
First, let’s take a look at the data structure aeEventLoop
corresponding to the Redis event-driven framework loop process. This structure is defined in the event-driven framework code ae.h and records information during the framework’s execution. It includes variables for two types of events:
- A pointer
*events
of typeaeFileEvent
representing IO events. The reason for naming itaeFileEvent
is that all IO events are identified by file descriptors. - A pointer
*timeEventHead
of typeaeTimeEvent
representing time events, which are events triggered periodically.
In addition, the aeEventLoop
structure also includes a pointer *fired
of type aeFiredEvent
. This is not a dedicated event type, but is instead used to record the file descriptor information of triggered events.
The following code shows the definition of the event loop structure in Redis:
typedef struct aeEventLoop {
...
aeFileEvent *events; // Array of IO events
aeFiredEvent *fired; // Array of triggered events
aeTimeEvent *timeEventHead; // Linked list that records time events
...
void *apidata; // Data related to API interface
aeBeforeSleepProc *beforesleep; // Function executed before entering the event loop
aeBeforeSleepProc *aftersleep; // Function executed after exiting the event loop
} aeEventLoop;
After understanding the aeEventLoop
structure, let’s take a look at how this structure is initialized, including the initialization of the IO event array and the time event linked list.
Initialization in the aeCreateEventLoop
Function
#
Since the Redis server starts running the event-driven framework loop process after initialization, the aeEventLoop
structure is initialized in the initServer
function of server.c by calling the aeCreateEventLoop
function. This function takes only one parameter, setsize
.
The following code shows the call to the aeCreateEventLoop
function in the initServer
function:
initServer() {
...
// Call the aeCreateEventLoop function to create the aeEventLoop structure
// and assign it to the el variable of the server structure
server.el = aeCreateEventLoop(server.maxclients+CONFIG_FDSET_INCR);
...
}
From here, we can see that the size of the setsize
parameter is determined by the maxclients
variable of the server structure and the macro definition CONFIG_FDSET_INCR
. The value of the maxclients
variable can be defined in the Redis configuration file redis.conf
. The default value is 1000. The size of the macro definition CONFIG_FDSET_INCR
is equal to the value of the macro definition CONFIG_MIN_RESERVED_FDS
plus 96. Both of these macro definitions are defined in the server.h file:
#define CONFIG_MIN_RESERVED_FDS 32
#define CONFIG_FDSET_INCR (CONFIG_MIN_RESERVED_FDS+96)
Now, you may have a question: What is the purpose of the setsize
parameter in the aeCreateEventLoop
function? This is related to the specific initialization operations performed by the aeCreateEventLoop
function.
Next, let’s take a look at the operations performed by the aeCreateEventLoop
function. It can be roughly divided into the following three steps.
Step 1: The aeCreateEventLoop
function creates a variable eventLoop
of type aeEventLoop
. Then, the function allocates memory for the member variables of eventLoop
, such as the IO event array and the array of triggered events, based on the setsize
parameter. In addition, the function initializes the member variables of eventLoop
to their initial values.
Step 2: The aeCreateEventLoop
function calls the aeApiCreate
function. The aeApiCreate
function encapsulates the IO multiplexing function provided by the operating system. Assuming Redis is running on the Linux operating system and the IO multiplexing mechanism is epoll, the aeApiCreate
function will call epoll_create
to create an epoll instance and create an array of epoll_event
structures, with the size of the array equal to the setsize
parameter.
Here, you should note that the aeApiCreate
function saves the created epoll instance descriptor and the array of epoll_event
structures in a variable of type aeApiState
named state
, as shown below:
typedef struct aeApiState { // Definition of the aeApiState structure
int epfd; // File descriptor of the epoll instance
struct epoll_event *events; // Array of epoll_event structures
} aeApiState;
int epfd; // Descriptor of the epoll instance
struct epoll_event *events; // Array of epoll_event structures, used to record the monitored events
} aeApiState;
static int aeApiCreate(aeEventLoop *eventLoop) {
aeApiState *state = zmalloc(sizeof(aeApiState));
...
// Save the epoll_event array in the aeApiState structure variable state
state->events = zmalloc(sizeof(struct epoll_event)*eventLoop->setsize);
...
// Save the descriptor of the epoll instance in the aeApiState structure variable state
state->epfd = epoll_create(1024);
}
Next, the aeApiCreate function assigns the state variable to apidata in the eventLoop. This way, the eventLoop structure has information about the epoll instance and the epoll_event array, which can be used to create and handle events based on epoll. I will explain in more detail later.
eventLoop->apidata = state;
**In the third step, the aeCreateEventLoop function initializes the mask for all network IO events' corresponding file descriptors as AE_NONE, indicating that no events are currently being monitored.**
Here is the main part of the aeCreateEventLoop function. You can take a look.
aeEventLoop *aeCreateEventLoop(int setsize) {
aeEventLoop *eventLoop;
int i;
// Allocate memory space for the eventLoop variable
if ((eventLoop = zmalloc(sizeof(*eventLoop))) == NULL) goto err;
// Allocate memory space for the IO events and fired events
eventLoop->events = zmalloc(sizeof(aeFileEvent)*setsize);
eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*setsize);
…
eventLoop->setsize = setsize;
eventLoop->lastTime = time(NULL);
// Set the time event list head to NULL
eventLoop->timeEventHead = NULL;
…
// Call the aeApiCreate function to actually call the IO multiplexing function provided by the operating system
if (aeApiCreate(eventLoop) == -1) goto err;
// Set the mask for all network IO events' corresponding file descriptors to AE_NONE
for (i = 0; i < setsize; i++)
eventLoop->events[i].mask = AE_NONE;
return eventLoop;
// Handling logic after initialization fails
err:
…
}
Well, from the execution flow of the aeCreateEventLoop function, we can see the following **two key points**:
- The size of the IO event array monitored by the event-driven framework is equal to the setsize parameter, which determines the number of clients connecting to the Redis server. So when you encounter the error "max number of clients reached" when connecting to Redis, you can modify the maxclients configuration parameter in the redis.conf file to increase the number of clients that the framework can monitor.
- When using the epoll mechanism of Linux, the initialization process in the framework loop will create an array of epoll_event structures through the aeApiCreate function and use the epoll_create function to create an epoll instance. These are the requirements for using the epoll mechanism. You can review the introduction to epoll in Lesson 9.
At this point, the framework can create and handle specific IO events and time events. So next, we will learn about IO events and their handling mechanisms.
IO Event Handling #
In fact, Redis’ IO events mainly include three types: readable events, writable events, and barrier events.
Among them, readable events and writable events are relatively easy to understand, they correspond to reading data from the client or writing data to the client. The main function of barrier events is to reverse the processing order of events. For example, by default, Redis returns the result to the client first. However, if it needs to write data to disk as soon as possible, Redis will use barrier events to adjust the order of writing data and replying to the client, first writing the data to disk and then replying to the client.
As I introduced to you in the previous lesson, in the Redis source code, the data structure of IO events is aeFileEvent struct, and the creation of IO events is completed through the aeCreateFileEvent function. The following code shows the definition of aeFileEvent struct, you can review it again:
typedef struct aeFileEvent {
int mask; // mask flag, including readable events, writable events, and barrier events
aeFileProc *rfileProc; // callback function for handling readable events
aeFileProc *wfileProc; // callback function for handling writable events
void *clientData; // private data
} aeFileEvent;
For the aeCreateFileEvent function, as we have learned in the previous lesson, it completes event registration through the aeApiAddEvent function. Next, let’s look at how it is executed from the code level, which can help us better understand how the event-driven framework listens to IO events based on the encapsulation of the epoll mechanism.
IO Event Creation #
First, let’s look at the prototype definition of aeCreateFileEvent function, as shown below:
int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask, aeFileProc *proc, void *clientData)
This function has 5 parameters: the event loop structure *eventLoop, the file descriptor fd corresponding to the IO event, the event type mask, the event processing callback function *proc, and the event private data *clientData.
Because the IO event array is in the event loop structure *eventLoop, and each element of this array corresponds to a file descriptor (e.g., a socket) associated with a listening event type and callback function.
The aeCreateFileEvent function will first obtain the IO event pointer variable *fe associated with the descriptor by the given file descriptor fd in the IO event array of the event loop, as shown below:
aeFileEvent *fe = &eventLoop->events[fd];
Next, the aeCreateFileEvent function will call the aeApiAddEvent function to add the event to listen to:
if (aeApiAddEvent(eventLoop, fd, mask) == -1)
return AE_ERR;
The aeApiAddEvent function actually calls the IO multiplexing function provided by the operating system to complete the event addition. Assuming that Redis is running on Linux using the epoll mechanism, aeApiAddEvent function will call the epoll_ctl function to add the event to listen. In fact, I already introduced the epoll_ctl function to you in Lesson 9. This function takes 4 parameters:
- epoll instance;
- the operation type to be executed (adding or modifying);
- the file descriptor to be listened;
- epoll_event type variable.
So, how does this calling process prepare the parameters required by the epoll_ctl function and complete the execution?
First, the epoll instance is the aeCreateEventLoop function I just introduced to you, which is created by calling the aeApiCreate function and saved in the apidata variable of the eventLoop structure, which is of type aeApiState. Therefore, the aeApiAddEvent function will first get this variable, as shown below:
static int aeApiAddEvent(aeEventLoop *eventLoop, int fd, int mask) {
aeApiState *state = eventLoop->apidata;
...
}
// Set the operation type based on the current mask
int op = eventLoop->events[fd].mask == AE_NONE ?
EPOLL_CTL_ADD : EPOLL_CTL_MOD;
// Create epoll_event variable ee
struct epoll_event ee = {0};
// Set the event types to be listened based on the mask
if (mask & AE_READABLE) ee.events |= EPOLLIN;
if (mask & AE_WRITABLE) ee.events |= EPOLLOUT;
ee.data.fd = fd;
// Call epoll_ctl to create the listening event
if (epoll_ctl(state->epfd,op,fd,&ee) == -1) return -1;
return 0;
Once an incoming connection request is received by the Redis server, the acceptTcpHandler() function is called to handle the client connection. This function accepts the client connection and creates a connected socket cfd
. Then, the acceptCommonHandler()
function is called with the newly created connected socket cfd
as an argument.
The acceptCommonHandler()
function calls the createClient()
function to create a new client. Within the createClient()
function, the aeCreateFileEvent()
function is called again to create a listening event for the connected socket cfd
, with the event type set to AE_READABLE
and the callback function set to readQueryFromClient()
.
OK, up to this point, the event-driven framework has added listening for a connected socket to the client. Once the client sends a request to the server, the framework will invoke the readQueryFromClient
function to handle the request. This allows client requests to be processed through the event-driven framework.
The following code shows the process of calling aeCreateFileEvent
in the createClient
function. Take a look:
client *createClient(int fd) {
// ...
if (fd != -1) {
// ...
// call aeCreateFileEvent to listen for read events, which correspond to client read/write requests, and use readQueryFromClient as the callback function to handle them
if (aeCreateFileEvent(server.el,fd,AE_READABLE,
readQueryFromClient, c) == AE_ERR)
{
close(fd);
zfree(c);
return NULL;
}
}
// ...
}
To help you understand the process of creating events from listening for client connection requests to listening for regular client read/write requests in the event-driven framework, I have created the following diagram for you to refer to:
After understanding the handling of read events in the event-driven framework, let’s take a look at the handling of write events.
Handling Write Events #
After a Redis instance receives a client request and processes the client command, it writes the data to be returned to the client output buffer. The following diagram shows the function call logic of this process:
Before entering the event handling function in the Redis event-driven framework, specifically when calling aeProcessEvents
in the main function aeMain
of the framework, to handle triggered events or timed events, the server.c file calls the beforeSleep
function to perform some tasks, including calling the handleClientsWithPendingWrites
function, which writes the data in the Redis server client buffer back to the client.
The following code is the main function aeMain
of the event-driven framework. Before calling aeProcessEvents
in this function, it calls the beforeSleep
function. Take a look:
void aeMain(aeEventLoop *eventLoop) {
eventLoop->stop = 0;
while (!eventLoop->stop) {
// if beforeSleep function is not NULL, call beforeSleep function
if (eventLoop->beforesleep != NULL)
eventLoop->beforesleep(eventLoop);
// after calling beforeSleep function, process events
aeProcessEvents(eventLoop, AE_ALL_EVENTS|AE_CALL_AFTER_SLEEP);
}
}
You need to know that the beforeSleep
function calls the handleClientsWithPendingWrites
function, which iterates through each client with pending writes and calls the writeToClient
function to write the data in the client output buffer back to the client. The following diagram shows this process:
However, if the data in the output buffer is not completely written, the handleClientsWithPendingWrites
function will call the aeCreateFileEvent
function to create a writable event and set the callback function sendReplyToClient
. The sendReplyToClient
function then calls the writeToClient
function to write the data.
The following code shows the basic flow of the handleClientsWithPendingWrites
function. Take a look:
int handleClientsWithPendingWrites(void) {
listIter li;
listNode *ln;
// ...
// get the list of clients with pending writes
listRewind(server.clients_pending_write,&li);
// iterate through each client with pending writes
while((ln = listNext(&li))) {
client *c = listNodeValue(ln);
// call writeToClient to write back the pending data in the current client output buffer
if (writeToClient(c->fd,c,0) == C_ERR) continue;
// if there are still pending replies
if (clientHasPendingReplies(c)) {
int ae_flags = AE_WRITABLE;
// create a listener for writable events and set the callback function
if (aeCreateFileEvent(server.el, c->fd, ae_flags,
sendReplyToClient, c) == AE_ERR)
{
// handle errors
// ...
}
}
}
}
So far, we have just learned the callback handling functions for read and write events. In order to process these events promptly, the aeMain
function of the Redis event-driven framework calls the aeProcessEvents
function in a loop to detect triggered events and call the corresponding callback functions for processing.
From the code of the aeProcessEvents
function, we can see that this function calls the aeApiPoll
function to query which descriptors are ready for listening. Once descriptors are ready, the aeProcessEvents
function calls the corresponding callback functions based on whether the event is readable or writable. The basic flow of the aeProcessEvents
function can be seen as follows:
int aeProcessEvents(aeEventLoop *eventLoop, int flags){
// ...
// call aeApiPoll to get ready descriptors
numevents = aeApiPoll(eventLoop, tvp);
// ...
for (j = 0; j < numevents; j++) {
aeFileEvent *fe = &eventLoop->events[eventLoop->fired[j].fd];
// ...
// if the triggered event is readable, call the read event callback function set when registering the event
if (!invert && fe->mask & mask & AE_READABLE) {
fe->rfileProc(eventLoop,fd,fe->clientData,mask);
fired++;
}
// if the triggered event is writable, call the write event callback function set when registering the event
if (fe->mask & mask & AE_WRITABLE) {
if (!fired || fe->wfileProc != fe->rfileProc) {
fe->wfileProc(eventLoop,fd,fe->clientData,mask);
fired++;
}
}
// ...
}
// ...
}
So far, we have covered the creation and handling of time events in the event-driven framework.
Time Event Handling #
Compared to IO events, which can be of types readable, writable, or barrier, and have different callback functions for different types of IO events, handling time events is relatively simple. In this section, we will learn about the definition, creation, callback function, and triggering of time events.
Time Event Definition #
First, let’s take a look at the struct definition of a time event. The code is as follows:
typedef struct aeTimeEvent {
long long id; // Time event ID
long when_sec; // Seconds timestamp when the event will be triggered
long when_ms; // Milliseconds timestamp when the event will be triggered
aeTimeProc *timeProc; // Callback function to be called when the event is triggered
aeEventFinalizerProc *finalizerProc; // Callback function to be called after the event has ended
void *clientData; // Private data associated with the event
struct aeTimeEvent *prev; // Pointer to the previous time event in the linked list
struct aeTimeEvent *next; // Pointer to the next time event in the linked list
} aeTimeEvent;
In the time event struct, the main variables include when_sec
and when_ms
, which represent the timestamps when the time event will be triggered in seconds and milliseconds, respectively. It also includes the callback function timeProc
to be called when the time event is triggered. Additionally, the time event struct contains prev
and next
pointers, indicating that time events are organized in the form of a linked list.
After understanding the definition of a time event struct, let’s move on to how time events are created.
Time Event Creation #
Similar to creating IO events using the aeCreateFileEvent
function, the function to create a time event is aeCreateTimeEvent
. The function prototype is as follows:
long long aeCreateTimeEvent(aeEventLoop *eventLoop, long long milliseconds, aeTimeProc *proc, void *clientData, aeEventFinalizerProc *finalizerProc)
Two important parameters in this function need our attention in order to understand time event handling. The first parameter is milliseconds
, which represents the duration from the current time when the time event will be triggered, measured in milliseconds. The second parameter is proc
, which represents the callback function to be called when the time event is triggered.
The execution logic of the aeCreateTimeEvent
function is not complex. It mainly creates a time event variable te
, initializes it, and inserts it into the time event linked list of the event loop structure eventLoop
. In this process, the aeCreateTimeEvent
function calls the aeAddMillisecondsToNow
function to calculate the specific trigger timestamp of the created time event based on the milliseconds
parameter, and assigns it to te
.
In fact, when Redis server is initialized, in addition to creating IO events for listening, it also calls the aeCreateTimeEvent
function to create time events. The code below shows the call to the aeCreateTimeEvent
function in the initServer
function:
initServer() {
...
// Create a time event
if (aeCreateTimeEvent(server.el, 1, serverCron, NULL, NULL) == AE_ERR){
... // Error message
}
...
}
From the code, we can see that the callback function called when the time event is triggered is serverCron
. So next, let’s take a look at the serverCron
function.
Time Event Callback Function #
The serverCron
function is implemented in the server.c
file. On one hand, it sequentially calls some functions to perform background tasks when the time event is triggered. For example, the serverCron
function checks if there is a process end signal, and if so, it executes the server shutdown operation. The serverCron
function calls the databaseCron
function to handle expired keys or perform rehashing operations. Refer to the code below:
...
// If a process end signal is received, execute the server shutdown operation
if (server.shutdown_asap) {
if (prepareForShutdown(SHUTDOWN_NOFLAGS) == C_OK) exit(0);
...
}
...
clientCron(); // Perform asynchronous operations for clients
databaseCron(); // Perform background operations for the database
...
On the other hand, the serverCron
function also periodically performs tasks at different frequencies. This is achieved by executing the run_with_period
macro.
The definition of the run_with_period
macro is as follows. This macro determines if the timestamp represented by the _ms_
parameter has been reached based on the hz
value defined in the Redis instance configuration file redis.conf
. Once it is reached, the serverCron
function can perform the corresponding tasks.
#define run_with_period(_ms_) if ((_ms_ <= 1000/server.hz) || !(server.cronloops%((_ms_)/(1000/server.hz))))
For example, the serverCron
function checks if there are any write errors in the AOF file every second. If there are, the serverCron
function calls the flushAppendOnlyFile
function to flush the cached data of the AOF file again. The code below shows this periodic task:
serverCron() {
...
// Execute once every second, check if there are write errors in AOF
run_with_period(1000) {
if (server.aof_last_write_status == C_ERR)
flushAppendOnlyFile(0);
}
...
}
If you want to learn more about periodic tasks, you can also carefully read the code blocks included in the serverCron
function defined by the run_with_period
macro.
Alright, now that we understand the callback function serverCron
called when the time event is triggered, let’s finally take a look at how time events are triggered and processed.
Time Event Triggering and Processing #
In fact, the detection and triggering of time events are relatively simple. The aeMain
function of the event-driven framework calls the aeProcessEvents
function in a loop to handle various events. At the end of the execution flow of the aeProcessEvents
function, it calls the processTimeEvents
function to process tasks that have reached their scheduled time.
aeProcessEvents() {
...
// Check if time events are triggered
if (flags & AE_TIME_EVENTS)
processed += processTimeEvents(eventLoop);
...
}
Regarding the processTimeEvents
function, its basic flow is to retrieve each event from the time event linked list one by one, and then determine whether the timestamp of the event is satisfied based on the current time. If satisfied, it calls the corresponding callback function of the event for processing. In this way, periodic tasks can be executed in the continuously looping aeProcessEvents
function.
The code below shows the basic flow of the processTimeEvents
function, you can take a look again:
static int processTimeEvents(aeEventLoop *eventLoop) {
...
te = eventLoop->timeEventHead; // Retrieve an event from the time event linked list
while(te) {
...
aeGetTime(&now_sec, &now_ms); // Get the current time
if (now_sec > te->when_sec || (now_sec == te->when_sec && now_ms >= te->when_ms)) // If the current time has reached the scheduled timestamp of the event
{
...
retval = te->timeProc(eventLoop, id, te->clientData); // Call the registered callback function to process
...
}
te = te->next; // Get the next time event
...
}
...
}
Summary #
In this lesson, I introduced two types of events in the Redis event-driven framework: IO events and time events.
For IO events, they can be further divided into readable, writable, and barrier events. Since readable and writable events are widely used in Redis for handling communication with clients, today we focused on these two types of IO events. After the Redis server creates a socket, it registers a readable event and uses the acceptTCPHandler callback function to handle client connection requests.
Once the server and client have established a connection, the server will listen for readable events on the connected socket and use the readQueryFromClient function to handle client read/write requests. Here, it is important to note that regardless of whether the client’s request is a read or write operation, the server needs to read the client’s request and process it. Therefore, the server registers a readable event on the connected socket.
When the server needs to write data back to the client, it registers a writable event in the event-driven framework and uses the sendReplyToClient function as the callback function to write the data from the buffer back to the client. I have summarized a table for you to review the correspondence between IO events, corresponding sockets, and callback functions.
For time events, they are mainly used to register periodically executed tasks in the event-driven framework for background processing by the Redis server. The callback function for time events is the serverCron function. You can further read about the specific tasks involved.
So, starting from Lesson 9, I have used three lessons to introduce the operation mechanism of the Redis event-driven framework. Essentially, the event-driven framework encapsulates the IO multiplexing mechanism provided by the operating system and adds support for time events. This is a very classic implementation of an event framework, and I hope you can learn and master it and apply it to your own system development. 在 Redis 中,根据不同的操作系统和他们所支持的 IO 多路复用机制,Redis 在调用 aeApiCreate、aeApiAddEvent 等函数时,会根据以下条件来确定具体调用哪个文件中的 IO 多路复用函数:
- 如果操作系统支持 epoll,Redis 会调用 ae_epoll.c 中定义的函数。
- 如果操作系统支持 evport,Redis 会调用 ae_evport.c 中定义的函数。
- 如果操作系统支持 kqueue,Redis 会调用 ae_kqueue.c 中定义的函数。
- 如果操作系统只支持 select,Redis 会调用 ae_select.c 中定义的函数。
每个操作系统对于 IO 多路复用的实现方式可能不同,Redis 根据操作系统的特性来选择合适的实现方式,以提高性能和效率。