31 Learning Dynamic Extension Functions From Module Implementation

31 Learning Dynamic Extension Functions from Module Implementation #

Redis itself already provides us with rich data types and data manipulation functions. Moreover, Redis implements a network framework based on IO multiplexing, data replication, fault recovery mechanisms, and data sharding clusters, which are typically the core functionalities needed by backend systems.

So, what should we do when we want to use the core functionalities already implemented by Redis, but also need to add some additional commands or data types?

In fact, Redis has provided the functionality of extension modules since version 4.0. These extension modules are loaded into Redis as dynamic link libraries (.so files), and we can add new functional modules based on Redis. These modules usually include new commands and data types. At the same time, the corresponding data of these data types will be stored in the Redis database, thus ensuring high-performance access to this data by the application.

Adding new functional modules is a problem that backend system development often encounters. In today’s lesson, I will show you how Redis implements the addition of a new functional module. Once you have mastered today’s content, you can refer to Redis’ implementation solution to add corresponding functional module extension frameworks to your own system, thereby increasing the flexibility of the system.

Next, let’s first understand the initialization operations of Redis extension module framework. Because the functionalities related to the Redis extension module framework are mainly defined and implemented in the [redismodule.h](https://github.com/redis/redis/tree/5.0/src/redismodule.h) and [module.c](https://github.com/redis/redis/tree/5.0/src/module.c) files, you can search for the data structures or functions that will be introduced next in these two files.

Initialization of the Module Framework #

In the execution flow of the main function of Redis, the moduleInitModulesSystem function (defined in module.c file) is called to initialize the extension module framework, as shown below:

int main(int argc, char **argv) {
   …
   moduleInitModulesSystem();
    …
}

The moduleInitModulesSystem function is mainly responsible for creating and initializing the data structures required for the extension module framework. Some important initialization operations include:

  • Creating a list to store the modules to be loaded, which corresponds to the loadmodule_queue member variable of the global variable server.
  • Creating a global hash table called modules to store the extension modules.
  • Calling the moduleRegisterCoreAPI function to register the core API.

The code for these operations is as follows:

void moduleInitModulesSystem(void) {
   ...
   server.loadmodule_queue = listCreate();
   modules = dictCreate(&modulesDictType,NULL);
   ...
   moduleRegisterCoreAPI();
   …
}

Let’s first take a look at what the moduleRegisterCoreAPI function does.

This function creates two hash tables, moduleapi and sharedapi, as member variables in the global variable server. These hash tables are used to store the APIs exposed by the modules and the APIs shared between modules, respectively. Then, the function calls the REGISTER_API macro to register the core API functions of the modules.

The following code shows part of the execution logic of the moduleRegisterCoreAPI function. You can see that it includes the registration of API functions such as Alloc, CreateCommand, ReplyWithLongLong, and ReplyWithError by calling the REGISTER_API macro.

void moduleRegisterCoreAPI(void) {
    server.moduleapi = dictCreate(&moduleAPIDictType,NULL);
    server.sharedapi = dictCreate(&moduleAPIDictType,NULL);
    REGISTER_API(Alloc);
    …
    REGISTER_API(CreateCommand);
    …
    REGISTER_API(ReplyWithLongLong);
    REGISTER_API(ReplyWithError);
    ...
}

These API functions are actually implemented by the Redis extension module framework itself, and we will use them when developing extension modules. For example, when we develop a new extension module, we will use the framework’s CreateCommand API to create new commands and the ReplyWithLongLong API to return results to the client.

Now, let’s take a closer look at how the REGISTER_API macro is implemented. It is implemented by the moduleRegisterApi function. The moduleRegisterApi function converts API functions starting with “RedisModule_” to ones starting with “RM_” and adds the API function to the global moduleapi hash table using the dictAdd function.

In this hash table, the key of each hash item is the name of the API, and the value is the corresponding function pointer. In this way, when we develop a module and need to use these API functions, we can look up the API name in the moduleapi hash table and obtain the corresponding function pointer for use.

The following code shows the definition of the REGISTER_API macro and the implementation of the moduleRegisterApi function:

#define REGISTER_API(name) \
    moduleRegisterApi("RedisModule_" #name, (void *)(unsigned long)RM_ ## name)

int moduleRegisterApi(const char *funcname, void *funcptr) {
    return dictAdd(server.moduleapi, (char*)funcname, funcptr);
}

In this way, we understand the work done during the initialization of the extension module framework. It mainly performs the initialization of the data structures required for the framework and adds the names and implementation functions of the APIs provided by the framework to the moduleapi hash table.

Next, let’s take a closer look at how to implement a module and see how it works.

Implementation and Workflow of Modules #

Let’s first look at a simple example of module implementation. Suppose we want to add a module called “helloredis”, which contains a command “hello” that returns the string “hello redis”.

In order to develop this new module, we need to develop two functions. One is the RedisModule_OnLoad function, which is called when the module is loaded to initialize the new module and register the module and command with the Redis module framework. The other is the implementation function of the new module’s specific functionality, which we name Hello_NewCommand.

Let’s first look at the process of initializing and registering the new module.

Initialization and Registration of New Module #

In the execution flow of the Redis entry point main function, after calling the moduleInitModulesSystem function to complete the initialization of the module framework, the main function will actually call the moduleLoadFromQueue function to load the extension modules.

The moduleLoadFromQueue function further calls the moduleLoad function, which loads the extension module based on the path of the module file and the parameters required by the module, as shown below:

void moduleLoadFromQueue(void) {
...
// Load the extension module
if (moduleLoad(loadmod->path,(void **)loadmod->argv,loadmod->argc) == C_ERR)
{...}
}

In the moduleLoad function, it searches for the “RedisModule_OnLoad” function in the module file we developed, and executes this function. Then, it calls the dictAdd function to add the successfully loaded module to the global hash table modules, as shown below:

int moduleLoad(const char *path, void **module_argv, int module_argc) {
...
// Search for the RedisModule_OnLoad function in the module file
onload = (int (*)(void *, void **, int))(unsigned long) dlsym(handle,"RedisModule_OnLoad");
...
// Execute the RedisModule_OnLoad function
if (onload((void*)&ctx,module_argv,module_argc) == REDISMODULE_ERR) {...}

...
dictAdd(modules,ctx.module->name,ctx.module); // Add the loaded module to the global hash table modules
}

I have drawn a diagram here to illustrate the process of the main function loading the new module, you can review it again.

From the process of loading the new module by the module framework in the main function, you can see that the module framework will search for the RedisModule_OnLoad function in the module file. RedisModule_OnLoad is a function that must be included in every new module and serves as the entry point for the module framework to load the new module. Through this function, we can register the new module command in the Redis command table, so that the new command can be used in Redis. The prototype of this function is as follows:

int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)

When we implement the RedisModule_OnLoad function, we need to use the API functions provided by the module framework introduced earlier.

First, we need to call the RedisModule_Init function (in the redismodule.h file) to register the new module. The function prototype is as follows:

static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver)

In the function prototype, the first parameter ctx is a variable of the RedisModuleCtx structure type, which records information such as the module pointer, the client executing the module command, and the runtime state. The second parameter name represents the name of the new module, and the third and fourth parameters represent the API version.

Then, for the “helloredis” module we mentioned earlier, we can call the RedisModule_Init function as follows to register the module:

if (RedisModule_Init(ctx,"helloredis",1,REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;

The specific registration process can be seen in the implementation of the RedisModule_Init function. The main work of this function can be divided into three steps. The first step is to set the RedisModule_GetApi function equal to the getapifuncptr function pointer in the RedisModuleCtx structure.

The second step is to call the REDISMODULE_GET_API macro to obtain the API functions provided by the extension module framework. This allows the new module to use the framework’s API.

Here, you need to pay attention to the definition of the REDISMODULE_GET_API macro, which actually uses the RedisModule_GetApi function pointer as follows:

#define REDISMODULE_GET_API(name) \
  RedisModule_GetApi("RedisModule_" #name, ((void **)&RedisModule_ ## name))

The RedisModule_GetApi function pointer is implemented through the REDISMODULE_API_FUNC macro. In this case, the REDISMODULE_API_FUNC macro sets its parameter as a function pointer, as shown below:

#define REDISMODULE_API_FUNC(x) (*x) // Set x as a function pointer

So, for the RedisModule_GetApi function pointer, it further points to the API function, whose parameters include the API function name and a pointer to the API function.

int REDISMODULE_API_FUNC(RedisModule_GetApi)(const char *, void *); // Set RedisModule_GetApi as a function pointer

Let’s take a look at the REDISMODULE_GET_API macro introduced earlier, as shown below:

int REDISMODULE_API_FUNC(RedisModule_GetApi)(const char *, void *); // Set RedisModule_GetApi as a function pointer

You will notice that this macro passes the parameter “name” to the RedisModule_GetApi function pointer, and the RedisModule_GetApi function pointer concatenates the parameter “name” with the string “RedisModule_”, creating the name of the API function in the module framework that starts with “RedisModule_”. This way, the pointer to the API function with the same name can be obtained.

Therefore, in the first and second steps of the RedisModule_Init function, RedisModule_GetApi is used to obtain the pointer to the API function.

Now, in the third step of the RedisModule_Init function, it calls the RedisModule_IsModuleNameBusy function to check if the registered module name already exists.

If the module already exists, it returns an error. If the module does not exist, it calls the RedisModule_SetModuleAttribs function to allocate a RedisModule structure for the new module and initialize the member variables of this structure. The RedisModule structure is used to record the properties of a module.

The following code shows part of the execution logic of the RedisModule_SetModuleAttribs function. You should pay attention to the moduleRegisterCoreAPI function I mentioned earlier, which sets the functions that start with “RedisModule_” to start with “RM_” at the initialization of the module framework. So when you see a function that starts with “RedisModule_”, you need to find the function that starts with “RM_” and has the same suffix in the module.c file.

void RM_SetModuleAttribs(RedisModuleCtx *ctx, const char *name, int ver, int apiver) {
    RedisModule *module;
    
    if (ctx->module != NULL) return;
    module = zmalloc(sizeof(*module));  // Allocate space for RedisModule structure
    module->name = sdsnew((char*)name); // Set module name
    module->ver = ver;  // Set module version
    …
    ctx->module = module; // Save the module pointer in the RedisModuleCtx variable that records the module's execution state
}

Okay, at this point, the initialization process for a new module in the RedisModule_Init function is completed. The following code also shows the main execution logic of the RedisModule_Init function for your review.

void *getapifuncptr = ((void**)ctx)[0];
RedisModule_GetApi = (int (*)(const char *, void *)) (unsigned long)getapifuncptr;
REDISMODULE_GET_API(Alloc);
…
REDISMODULE_GET_API(CreateCommand);
…
REDISMODULE_GET_API(ListPush);
REDISMODULE_GET_API(ListPop);
…
REDISMODULE_GET_API(CreateString);
…
// Check if there is a module with the same name
if (RedisModule_IsModuleNameBusy && RedisModule_IsModuleNameBusy(name)) return REDISMODULE_ERR;
RedisModule_SetModuleAttribs(ctx,name,ver,apiver); // If there is no module with the same name, initialize the data structure of the module
return REDISMODULE_OK;

Actually, from the code, you can find that the function RedisModule_Init obtains many API functions for key-value pair operations from the framework when initializing a new module, such as the Push and Pop operations for List, and the creation of a String, etc. You can further read the RedisModule_Init function to understand the APIs that a new module can obtain.

So, after we call the RedisModule_Init function to register and initialize the new module, we can use the RedisModule_CreateCommand function to register new commands for the module. Now, let’s take a look at the execution process of this function.

Registering a New Command #

For the new module we developed, we need to add a new command called “hello”. This is mainly done by calling the RedisModule_CreateCommand function in the RedisModule_OnLoad function. First, take a look at the following code, which registers the new command.

int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
…
if (RedisModule_CreateCommand(ctx, "hello", Hello_NewCommand, "fast", 0, 0, 0) == REDISMODULE_ERR)
return REDISMODULE_ERR;
…}

From the code, you can see that the RedisModule_CreateCommand function’s parameters include the name of the new command “hello”, the implementation function Hello_NewCommand for this command, and the property tag “fast” for this command.

Now, let’s take a look at the execution process of RedisModule_CreateCommand, as I mentioned earlier, the actual implementation function it corresponds to is RM_CreateCommand, which starts with “RM_”.

The prototype of the RM_CreateCommand function is shown below, with its second, third, and fourth parameters corresponding to the name of the new command, the command’s implementation function, and the command’s flag, respectively.

int RM_CreateCommand(RedisModuleCtx *ctx, const char *name, RedisModuleCmdFunc cmdfunc, const char *strflags, int firstkey, int lastkey, int keystep)

The main purpose of the RM_CreateCommand function is to create a variable of the RedisModuleCommandProxy structure type, which is similar to a proxy command for the new command. It itself records the implementation function for the new command, and at the same time, it creates a member variable rediscmd of the redisCommand structure type.

Here you need to pay attention that in the Redis source code, a variable of type redisCommand corresponds to a command in the Redis command table. When Redis receives a command from the client, it will look up the command name and the corresponding redisCommand variable in the command table. The member variable proc of the redisCommand structure corresponds to the implementation function of the command.

struct redisCommand {
    char *name;  // Command name
    redisCommandProc *proc;  // Implementation function of the command
    ...
}

In the previously mentioned variable cp, it creates a member variable rediscmd of type redisCommand and sets its proc variable to the RedisModuleCommandDispatcher function.

Then, the RM_CreateCommand function adds rediscmd to the Redis command table. In this way, when the client sends the new command, Redis will first find that the execution function for the new command is RedisModuleCommandDispatcher in the command table, and then execute this RedisModuleCommandDispatcher function. The RedisModuleCommandDispatcher function will then actually call the implementation function of the new module command.

The figure below shows the relationship between the proxy command and the new module command when the RM_CreateCommand function adds the proxy command. You can take a look. Redis Command Structure

The code below also shows the basic execution logic of creating the proxy command and adding it to the Redis command table in the RM_CreateCommand function. You can review it again.

struct redisCommand *rediscmd;
RedisModuleCommandProxy *cp;  // Create a RedisModuleCommandProxy structure variable
sds cmdname = sdsnew(name);  // Name of the new command
cp = zmalloc(sizeof(*cp));
cp->module = ctx->module;  // Record the module corresponding to the command
cp->func = cmdfunc;  // Implementation function of the command
cp->rediscmd = zmalloc(sizeof(*rediscmd));  // Create a redisCommand structure corresponding to the command in the Redis command table
cp->rediscmd->name = cmdname;  // Command name in the command table
cp->rediscmd->proc = RedisModuleCommandDispatcher;  // Implementation function of the command in the command table
dictAdd(server.commands, sdsdup(cmdname), cp->rediscmd);

So, we need to call the RedisModule_CreateCommand function in the RedisModule_OnLoad function when developing a new module, which is the second step to be completed. This will register the new command in the Redis command table.

Now, you can take a look at the code below, which shows the content of the new module we developed so far. This completes the development of a simple RedisModule_OnLoad function.

int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
  // Initialize the module
  if (RedisModule_Init(ctx, "helloredis", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR)
    return REDISMODULE_ERR;

  // Register the command
  if (RedisModule_CreateCommand(ctx, "hello", Hello_NewCommand, "fast", 0, 0, 0) == REDISMODULE_ERR)
    return REDISMODULE_ERR;

  return REDISMODULE_OK;
}

Next, we need to develop the actual implementation function for the new command.

Developing the Implementation Function for the New Command #

The implementation function for the new command is developed to implement the specific functionality of the new module. In the example we just mentioned, the new module “helloredis” has a command “hello”, which simply returns the string “hello redis”.

When we registered the implementation function for the new command using the RedisModule_CreateCommand function, we registered the function Hello_NewCommand. So, here we need to implement this function.

The code below shows the logic of the Hello_NewCommand function. As you can see, it simply calls the RedisModule_ReplyWithString function to return the string “hello redis” to the client.

int Hello_NewCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
    return RedisModule_ReplyWithString(ctx, "hello redis");
}

Furthermore, from the code, you can also see that our developed module can call API functions provided by the extension module framework to accomplish certain functionalities. For example, in the previous code, the Hello_NewCommand function calls the RedisModule_ReplyWithString API function provided by the framework to return a string result to the client.

Alright, at this point, we have completed the development of a simple new module. This process includes developing the RedisModule_OnLoad function to initialize the module and register the new command, as well as implementing the Hello_NewCommand function to implement the module’s functionality.

Finally, let’s take a look at how Redis executes the new module command.

Execution of the New Module Command #

As I mentioned earlier, when the main function is executed, it calls the moduleLoadFromQueue function to load the extension module. Once the module is loaded, it can start accepting the new command from it.

As I explained in Lecture 14, the execution of a command follows a certain process, and it also applies to the execution of the new command from the extension module. When a command from an extension module is received, the processCommand function is called, and this function searches for the received command in the command table. If the command is found, the processCommand function calls the call function to execute the command.

The call function will execute the function pointed to by the proc pointer in the redisCommand structure corresponding to the command sent by the client. Here’s an example:

void call(client *c, int flags) {
    ...
    c->cmd->proc(c);
    ...
}

Note that when registering the new command with the RM_CreateCommand function, the corresponding function registered in the command table is RedisModuleCommandDispatcher. So, when the new module command is received, the RedisModuleCommandDispatcher function is executed.

The RedisModuleCommandDispatcher function first obtains the variable cp, which represents the proxy command structure I mentioned earlier, and then calls the func member variable of cp. The func member variable was assigned the actual implementation function of the new command when the RM_CreateCommand function was executed. In this way, the new module command can be executed properly through the RedisModuleCommandDispatcher function.

The code below shows the basic logic of the RedisModuleCommandDispatcher function. Please take a look:

void RedisModuleCommandDispatcher(client *c) {
  RedisModuleCommandProxy *cp = (void*)(unsigned long)c->cmd->getkeys_proc;
  ...
  cp->func(&ctx, (void**)c->argv, c->argc);
  ...
}

Alright, now we understand how the execution of the new module command is accomplished through the implementation function of the proxy command, RedisModuleCommandDispatcher. In this way, we have a clear understanding of the whole process from the development of the module itself to the execution of module commands.

Summary #

In today’s lesson, I introduced you to the working mechanism of the Redis extension module framework. Using a simple extension module as an example, I explained the initialization of the extension module framework, the initialization of a new module, and the process of registering and executing new commands. In this process, there are three key points that you need to focus on.

Firstly, the program for adding a new module must include the RedisModule_OnLoad function. This is because when the module framework loads the module, it uses the dynamic link library function dlsym to search for the RedisModule_OnLoad function in the dynamic link file (so file) compiled after adding the module, and then executes this function. Therefore, when developing an extension module, we need to use the RedisModule_Init function to initialize the module in the RedisModule_OnLoad function, and use the RedisModule_CreateCommand function to register commands.

Secondly, the extension module framework does not directly add the implementation function of the new command to the Redis command table. Instead, it sets the execution function of the new command to RedisModuleCommandDispatcher, and then the RedisModuleCommandDispatcher function executes the actual implementation function of the new command.

Thirdly, the extension module framework itself encapsulates many existing Redis operational functionalities using API functions starting with “RM_”. For example, operations on different data types and replying to clients with different types of results. This facilitates the reuse of existing Redis functionalities when developing new modules. You can further read the module.c file to understand the specific API functions provided by the extension framework.

Finally, the three points summarized above are helpful for us in developing extension modules and understanding their operational mechanisms. They also provide a reference implementation for us to develop our own extension module frameworks. I hope you can grasp them well.

One question per lesson #

Have you used any Redis extension modules, or have you developed your own extension modules? Feel free to share your experience in the comments.