10 Did Redis Implement the Reactor Pattern in the Event-Driven Framework #

Today, let’s talk about how Redis implements the Reactor model.

When you are preparing for Redis interview questions, you may often come across this classic question: Does Redis’s network framework implement the Reactor model? This seems like a simple “yes/no” question, but if you want to provide a satisfactory answer to the interviewer, it will test both your high-performance network programming foundation and your understanding of Redis code.

If I were to answer this question, I would divide it into two parts: first, explaining what the Reactor model is, and second, explaining how Redis code implementation corresponds to the Reactor model. This way, it not only demonstrates my understanding of network programming but also showcases a deep exploration of the Redis source code, which would impress the interviewer.

In reality, the Reactor model is an important technical solution for implementing high-performance network systems to handle concurrent requests. Mastering the design principles and implementation methods of the Reactor model not only helps with interview questions but also guides you in designing and implementing your own highly concurrent systems. You won’t be at a loss when dealing with thousands of network connections.

So, in today’s lesson, I will first introduce you to the Reactor model and then teach you how to implement it. Redis’s code implementation offers a good reference example, so I will use key functions and processes in the Redis code to explain the implementation of the Reactor model to you. However, before diving into the Reactor model, you should review the previous lesson where I introduced the I/O multiplexing mechanism epoll, as it forms the foundation for today’s lesson.

How the Reactor Model Works #

Okay, first let’s take a look at what the Reactor model is.

In fact, the Reactor model is a programming model used by network servers to handle high-concurrency network IO requests . I summarize the characteristics of this model with two “threes”:

• Three types of processing events: connection event, write event, and read event;
• Three key roles: reactor, acceptor, and handler.

So, how does the Reactor model handle high-concurrency requests based on these three types of events and three roles? Let’s take a closer look.

Event Types and Key Roles #

Let’s start by looking at the relationship between these three event types and the Reactor model.

In fact, the Reactor model deals with the interaction process between clients and servers, and these three types of events correspond to the pending events triggered by different types of requests in the server during the interaction process:

• When a client wants to interact with the server, the client sends a connection request to the server to establish a connection, which corresponds to a connection event in the server.
• Once the connection is established, the client sends a read request to the server to read data. When the server handles the read request, it needs to write data back to the client, which corresponds to a write event in the server.
• Regardless of whether the client sends a read or write request to the server, the server needs to read the request content from the client. So here, reading the read or write request corresponds to a read event in the server.

The figure below shows the correspondence between different types of requests in the interaction between clients and servers and the events of the Reactor model. Take a look.

Okay, after understanding the three types of events in the Reactor model, you may still have a question: who handles these three types of events?

This is the role of the three key roles in the model:

• First, the acceptor handles the connection event and is responsible for accepting connections. After the acceptor receives a connection, it creates a handler for handling subsequent read and write events on the network connection.
• Second, the handler handles the read and write events.
• Finally, in a high-concurrency scenario, connection events and read/write events may occur simultaneously. Therefore, we need a role to listen to and distribute events, which is the reactor role. When a connection request occurs, the reactor hands over the connection event to the acceptor for processing. When a read/write request occurs, the reactor hands over the read/write event to the handler for processing.

The figure below shows the relationship between these three roles and their relationship with events. Take a look.

In fact, these three roles are abstractions of the functions to be implemented in the Reactor model. When we develop a network framework for servers following the Reactor model, we need to implement the logic of reactor, acceptor, and handler in the code modules of the functionality.

Now, we already know that these three roles interact based on event listening, dispatching, and handling. So, how do we implement the interaction among them in programming? This requires an event-driven framework.

Event-Driven Framework #

The so-called event-driven framework is the overall control logic that needs to be implemented when implementing the Reactor model. Simply put, the event-driven framework consists of two parts: event initialization, and event capture, dispatch, and handling main loop.

Event initialization is executed when the server program starts. Its main purpose is to create the types of events that need to be listened for, and the handlers corresponding to these events. Once the server completes initialization, the event initialization is also completed accordingly, and the server program needs to enter the main loop of event capture, dispatch, and handling.

When developing code, we usually use a while loop as the main loop. Then in this main loop, we need to capture the occurring events, determine the event types, and call the event handler created during initialization to actually handle the events based on the event types.

For example, when a connection event occurs, the server program needs to call the acceptor processing function to create a connection with the client. And when a read event occurs, it means that a read or write request has been sent to the server, so the server program needs to call the specific request processing function to read the request content from the client connection, thus completing the handling of the read event. You can refer to the figure below, which shows the basic execution process of the event-driven framework:

So, by now, you should have understood the basic working mechanism of the Reactor model: different types of requests from clients trigger connection, read, and write events in the server, and these three types of events are processed by the three roles of reactor, acceptor, and handler. Then these three roles interact and handle events through the event-driven framework.

Therefore, it can be seen that the key to implementing a Reactor model is to implement the event-driven framework. So, how do we develop and implement an event-driven framework?

Redis provides a concise but effective reference implementation, which is very worth learning and can be used for developing your own network systems. Now, let’s learn about the implementation of the Reactor model in Redis together.

Implementation of Reactor Model in Redis #

First, we need to know that Redis’ network framework implements the Reactor model and has independently developed an event-driven framework. The Redis code files corresponding to this framework are ae.c and the corresponding header file is ae.h.

As we have learned before, the implementation of the event-driven framework relies on the definition of events, as well as a series of operations such as event registration, capturing, dispatching, and handling. Of course, for the entire framework to run continuously and respond to events that occur, it also needs a main loop.

Therefore, from the ae.h header file, we can see that Redis defines the data structure of events, the main loop function of the framework, the event capturing and dispatching function, and the event and handler registration functions to implement the event-driven framework. So next, let’s learn about these functions step by step.

Data Structure Definition of Events: Taking aeFileEvent as an Example #

First, we need to clarify that in the implementation of the event-driven framework in Redis, the data structure of events is a key element that associates event types with event handling functions. Redis’ event-driven framework defines two types of events: IO events and timer events, which correspond to network requests sent by clients and periodic operations of Redis itself.

This means that the data structure definitions for different types of events differ. However, since in this lesson we mainly focus on the overall design and implementation of the event framework, I will provide detailed explanations about the differences and specific handling of different types of events in the next lesson. In today’s lesson, in order for you to understand the role of event data structures in the framework, I will take the IO event aeFileEvent as an example to introduce its data structure definition.

aeFileEvent is a structure that defines 4 member variables: mask, rfileProc, wfileProc, and clientData, as shown below:

typedef struct aeFileEvent {
    int mask; /* one of AE_(READABLE|WRITABLE|BARRIER) */
    aeFileProc *rfileProc;
    aeFileProc *wfileProc;
    void *clientData;
} aeFileEvent;

• mask is used to represent the event type mask. For network communication events, there are mainly three types of events: AE_READABLE, AE_WRITABLE, and AE_BARRIER. When dispatching events, the framework relies on the event type defined in the structure.
• rfileProc and wfileProc are pointers to the handling functions of the AE_READABLE and AE_WRITABLE event types, which are the handlers in the Reactor model. After dispatching events, the functions defined in the structure need to be called for event handling.
• The last member variable, clientData, is a pointer to client-specific data.

In addition to the data structure of events, as mentioned earlier, Redis also defines the main functions that support the operation of the framework in the ae.h file, including the aeMain function for the main loop, the aeCreateFileEvent function for event and handler registration, and the aeProcessEvents function for event capturing and dispatching. Their prototype definitions are as follows:

void aeMain(aeEventLoop *eventLoop);
int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask, aeFileProc *proc, void *clientData);
int aeProcessEvents(aeEventLoop *eventLoop, int flags);

These three functions are implemented in the corresponding ae.c file. Next, I will provide a detailed explanation of the main logic and key processes of these three functions.

Main Loop: aeMain Function #

Let’s start with the aeMain function.

The logic of the aeMain function is simple. It uses a loop to continuously check the stop flag of the event loop. If the stop flag of the event loop is set to true, the main loop for event capturing, dispatching, and handling will stop. Otherwise, the main loop will keep executing. The main body code of the aeMain function is as follows:

void aeMain(aeEventLoop *eventLoop) {
    eventLoop->stop = 0;
    while (!eventLoop->stop) {
        …
        aeProcessEvents(eventLoop, AE_ALL_EVENTS|AE_CALL_AFTER_SLEEP);
    }
}

Now you might be wondering, where is the aeMain function called?

According to the programming specifications of the event-driven framework, the main loop of the framework starts executing after the server program is initialized. Therefore, if we shift our focus to the server initialization function of Redis, we will find that the main function of the server program calls the aeMain function to start executing the event-driven framework after completing the initialization of the Redis server. If you want to see the specific implementation of the main function, it is in the server.c file, which we introduced in Lesson 8. server.c is mainly used for server initialization and executing the overall server control flow.

However, since the aeMain function contains the main loop of the event-driven framework, how are events captured, dispatched, and handled in the main loop? This is accomplished by the aeProcessEvents function.

Event Capturing and Dispatching: aeProcessEvents Function #

The aeProcessEvents function implements the main functionalities of event capturing, event type determination, and invoking specific event handling functions, thereby achieving event handling.

From the structure of the aeProcessEvents function, we can see that it has three if branches, as shown below:

int aeProcessEvents(aeEventLoop *eventLoop, int flags)
{
    int processed = 0, numevents;
 
    /* If no events need processing, return immediately */
    if (!(flags & AE_TIME_EVENTS) && !(flags & AE_FILE_EVENTS)) return 0;
    /* If there are IO events or urgent time events that need to be processed, start processing */
    if (eventLoop->maxfd != -1 || ((flags & AE_TIME_EVENTS) && !(flags & AE_DONT_WAIT))) {
       …
    }
    /* Check if there are time events and process them by calling the processTimeEvents function */
    if (flags & AE_TIME_EVENTS)
        processed += processTimeEvents(eventLoop);
    /* Return the number of processed files or time */
    return processed; 
}

These three branches correspond to the following three situations:

1. Case 1: Neither timer events nor network events need to be processed.
2. Case 2: There are IO events or time events that need to be urgently processed.
3. Case 3: There are only normal time events.

For the first situation, since there are no events to be processed, the aeProcessEvents function will directly return to the main loop of aeMain to start the next round of the loop. For the third situation, which occurs when only normal time events occur, the aeMain function will call the function dedicated to processing time events, processTimeEvents, to process the time events. Now let’s take a look at the second scenario.

Firstly, when this situation occurs, Redis needs to capture the network events that occur and process them accordingly. In the Redis source code, we can analyze that in this case, the aeApiPoll function is called to capture events as shown below:

int aeProcessEvents(aeEventLoop *eventLoop, int flags){
   ...
   if (eventLoop->maxfd != -1 || ((flags & AE_TIME_EVENTS) && !(flags & AE_DONT_WAIT))) {
       ...
       // Call aeApiPoll function to capture events
       numevents = aeApiPoll(eventLoop, tvp);
       ...
    }
    ...
}

So, how does aeApiPoll capture events?

In fact, Redis relies on the IO multiplexing mechanism provided by the underlying operating system to capture events and check for new connections and read/write events. In order to adapt to different operating systems, Redis has unified the implementation of network IO multiplexing functions for different operating systems, and the encapsulated code is implemented in the following four files:

• ae_epoll.c, corresponding to the epoll IO multiplexing function on Linux
• ae_evport.c, corresponding to the evport IO multiplexing function on Solaris
• ae_kqueue.c, corresponding to the kqueue IO multiplexing function on macOS or FreeBSD
• ae_select.c, corresponding to the select IO multiplexing function on Linux (or Windows)

In this way, with these encapsulated codes, Redis can call IO multiplexing APIs on different operating systems through a unified interface.

However, at this point, you may still not fully understand the specific operations of Redis’ encapsulation. Therefore, here, I will take the most commonly used Linux operating system on the server side as an example to explain how Redis encapsulates the IO multiplexing API provided by Linux.

Firstly, Linux provides the epoll_wait API to detect network IO events that occur in the kernel. In the ae_epoll.c file, the aeApiPoll function encapsulates the call to epoll_wait.

The encapsulation code is as follows. You can see that the aeApiPoll function directly calls the epoll_wait function and saves the event information returned by epoll:

static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) {
    …
    // Call epoll_wait to get the listened events
    retval = epoll_wait(state->epfd,state->events,eventLoop->setsize,
            tvp ? (tvp->tv_sec*1000 + tvp->tv_usec/1000) : -1);
    if (retval > 0) {
        int j;
        // Get the number of listened events
        numevents = retval;
        // For each event, process it
        for (j = 0; j < numevents; j++) {
             # Save the event information
        }
    }
    return numevents;
}

To help you understand more clearly how the event-driven framework ultimately calls epoll_wait, I have also included a diagram to show how the entire call chain works and is implemented.

OK, now we have seen in the aeMain function that the aeProcessEvents function is called and used for the basic processing logic of capturing and distributing events.

So, which function handles the events specifically? This is related to the aeCreateFileEvents function in the framework.

Event Registration: aeCreateFileEvent function #

As we know, when Redis starts, the main function of the server program calls the initServer function for initialization. During the initialization process, aeCreateFileEvent is called by the initServer function to register the events to be listened for and the corresponding event handling functions.

Specifically, during the execution of the initServer function, aeCreateFileEvent will be called by the initServer function for each network event on each IP port based on the number of enabled IP ports. It creates a listener for the AE_READABLE event and registers the AE_READABLE event handling function, which is the acceptTcpHandler function. The process is shown in the following figure:

As you can see, the AE_READABLE event represents a client’s network connection event, and the corresponding handler is the function that accepts TCP connection requests. The following code snippet shows a partial segment of the call to aeCreateFileEvent in the initServer function:

void initServer(void) {
    …
    for (j = 0; j < server.ipfd_count; j++) {
        if (aeCreateFileEvent(server.el, server.ipfd[j], AE_READABLE,
            acceptTcpHandler,NULL) == AE_ERR)
            {
                serverPanic("Unrecoverable error creating server.ipfd file event.");
            }
  }
  …
}

So how does aeCreateFileEvent register events and handling functions? This is related to the Redis encapsulation of the underlying IO multiplexing functions I mentioned earlier. Let’s continue with the example of the Linux system.

Firstly, Linux provides the epoll_ctl API to add new events to be observed. Redis then encapsulates the aeApiAddEvent function based on this, which in turn calls epoll_ctl to register the desired events to be listened for and their corresponding handling functions. When aeProceeEvents captures actual events, it calls the registered function to process the events.

Alright, now we have a complete understanding of the three key functions in Redis that implement the event-driven framework: aeMain, aeProcessEvents, and aeCreateFileEvent. When you want to implement an event-driven framework, the design principles of Redis are very valuable for reference.

Finally, let’s briefly review. When implementing an event-driven framework, you first need to implement a main loop function (corresponding to aeMain) that is responsible for running the framework continuously. Secondly, you need to write an event registration function (corresponding to aeCreateFileEvent) to register the events to be listened for and their corresponding handling functions. Only by registering events and handling functions can the corresponding functions be called to process the events when they occur.

Finally, you need to write an event listener and dispatcher function (corresponding to aeProcessEvents), which is responsible for calling the underlying operating system functions to capture network connection, read, and write events and dispatch them to different handling functions for further processing.

Summary #

Redis has always been known for its single-threaded architecture, which means that a single thread can only process requests from a single client. However, in practice, we can see that Redis can interact with hundreds or thousands of clients simultaneously. This is because Redis is based on the Reactor model, which implements a high-performance network framework. Using an event-driven framework, Redis can continuously capture, dispatch, and handle network connection, data reading, and writing events generated by clients through a loop.

To help you understand the implementation of the Redis event-driven framework from a code perspective, I have summarized a table that lists the main functions and functionalities of the Redis event-driven framework, the corresponding C files they belong to, and where these functions are called in the Redis code structure. You can use this table to reinforce your understanding of the event-driven framework that we learned in today’s class.

Finally, I would like to emphasize again that in this class, our main focus was on the basic workflow of the event-driven framework, and we used the client connection event as an example to explain the key steps of the framework, including the main loop, event capture and dispatch, and event registration. The events that the Redis event-driven framework listens and handles also include client requests, server-side data writing, and periodic operations, which will be the main topics we will learn together in my next class.

One Question per Lesson #

In this lesson, we learned about the Reactor model. Apart from Redis, do you know of any other software systems that use the Reactor model?