18 How to Generate and Parse Rdb Files

18 How to Generate and Parse RDB Files #

Starting from today’s lesson, we will enter a new module, which is the reliability assurance module. In this module, I will first introduce the implementation of Redis data persistence, including the methods to generate Redis memory snapshots in RDB files and the recording and rewriting of AOF logs. Understanding this part will enable you to grasp the format of RDB files, learn how to create database mirrors, and further understand the impact of AOF log rewriting on Redis performance.

In addition, I will also introduce the code implementation of Redis master-slave replication, sentinel working mechanism, and failover process. As we know, master-slave replication is an important mechanism for ensuring the reliability of distributed data systems, and Redis provides us with a very classic implementation. Therefore, by learning this part, you will be able to master some key operations and considerations in the data synchronization process to avoid pitfalls.

Alright, let’s start with generating RDB files in today’s lesson. Now, let’s first understand the entry function for RDB creation and where these functions are called.

Entry Functions and Triggers for RDB Creation #

There are three functions in the Redis source code used to create RDB files, all of which are implemented in the rdb.c file. Let’s take a closer look at them:

  • rdbSave function

This is the entry function for creating an RDB file on the local disk in Redis server. It corresponds to the SAVE command in Redis and is called in the implementation function saveCommand (in the rdb.c file). The rdbSave function ultimately calls the rdbSaveRio function (also in the rdb.c file) to actually create the RDB file. The execution logic of the rdbSaveRio function reflects the format and generation process of the RDB file, which I will explain to you later.

  • rdbSaveBackground function

This is the entry function for creating an RDB file on the local disk in Redis server using a background child process. It corresponds to the BGSAVE command in Redis and is called in the implementation function bgsaveCommand (in the rdb.c file). This function calls fork to create a child process, which then calls the rdbSave function to continue creating the RDB file. The parent process, which is the main thread itself, can continue to process client requests.

The code below demonstrates the process of creating a child process in the rdbSaveBackground function. Take a look. I also introduced the use of fork to you in Lesson 12, so you can review it again.

int rdbSaveBackground(char *filename, rdbSaveInfo *rsi) {
...
if ((childpid = fork()) == 0) {  // Code branch executed by the child process
   ...
   retval = rdbSave(filename,rsi);  // Call rdbSave function to create the RDB file
   ...
   exitFromChild((retval == C_OK) ? 0 : 1);  // Child process exits
} else {
   ...  // Code branch executed by the parent process
}
}
  • rdbSaveToSlavesSockets function

This is the entry function for creating an RDB file in Redis server when using diskless transmission for master-slave replication. It is called by the startBgsaveForReplication function (in the replication.c file). The startBgsaveForReplication function is called by the syncCommand function and replicationCron function in the replication.c file, corresponding to the execution of the master-slave replication command and the periodic checking of the master-slave replication status triggering RDB generation.

Similar to the rdbSaveBackground function, the rdbSaveToSlavesSockets function also creates a child process through fork to generate the RDB. However, unlike the rdbSaveBackground function, the rdbSaveToSlavesSockets function directly sends the binary data of the RDB file as a byte stream over the network to the slave nodes.

In order for the slave nodes to recognize the RDB content used for data synchronization, the rdbSaveToSlavesSockets function calls the rdbSaveRioWithEOFMark function (in the rdb.c file) to add identifying strings before and after the binary data of the RDB file, as shown in the following figure:

The code below also shows the basic execution logic of the rdbSaveRioWithEOFMark function. As you can see, besides writing the identifying strings before and after, it also calls the rdbSaveRio function to actually generate the RDB content.

int rdbSaveRioWithEOFMark(rio *rdb, int *error, rdbSaveInfo *rsi) {
...
getRandomHexChars(eofmark,RDB_EOF_MARK_SIZE); // Generate a random 40-byte hexadecimal string and store it in eofmark. The value of the macro RDB_EOF_MARK_SIZE is 40
if (rioWrite(rdb,"$EOF:",5) == 0) goto werr;  // Write "$EOF"
if (rioWrite(rdb,eofmark,RDB_EOF_MARK_SIZE) == 0) goto werr; // Write the 40-byte hexadecimal string eofmark
if (rioWrite(rdb,"\r\n",2) == 0) goto werr; // Write "\r\n"
if (rdbSaveRio(rdb,error,RDB_SAVE_NONE,rsi) == C_ERR) goto werr; // Generate the RDB content
if (rioWrite(rdb,eofmark,RDB_EOF_MARK_SIZE) == 0) goto werr; // Write the 40-byte hexadecimal string eofmark again
...
}

Alright, now that you understand the three entry functions for creating RDB files, you’ve also seen three triggers for RDB file creation: the execution of the SAVE command, the execution of the BGSAVE command, and master-slave replication. So, in addition to these three triggers, where else in the Redis source code does RDB file creation occur?

In fact, since the rdbSaveToSlavesSockets function is only called during master-slave replication, we can understand the other triggers for RDB file creation by searching for the rdbSave and rdbSaveBackground functions in the Redis source code.

By searching, we can find that the rdbSave function is also called in the flushallCommand function (in the db.c file) and the prepareForShutdown function (in the server.c file). This means that Redis creates an RDB file when executing the FLUSHALL command and when shutting down normally.

As for the rdbSaveBackground function, in addition to being called when executing the BGSAVE command, it is also called by the startBgsaveForReplication function when master-slave replication uses disk file transfer. Additionally, the serverCron function that runs periodically during Redis server execution also calls the rdbSaveBackground function to create RDB files.

To help you grasp the overall situation of RDB file creation, I have created the following diagram, which shows the function call relationships for RDB file creation in the Redis source code. Take a look.

Okay, up until this point, you can see that the actual function that generates the RDB file is rdbSaveRio. Next, let’s take a look at the execution process of the rdbSaveRio function. At the same time, I will also explain how the RDB file is structured.

How is an RDB file generated? #

Before understanding how the rdbSaveRio function specifically generates an RDB file, you need to first understand the basic components of an RDB file. This way, you can understand the execution logic of the rdbSaveRio function based on the components of an RDB file.

An RDB file mainly consists of three parts:

  • File header: This part contains information such as the magic number of Redis, the RDB version, the Redis version, the creation time of the RDB file, and the memory size occupied by key-value pairs.
  • File data: This part contains all the actual key-value pairs in the Redis database.
  • File footer: This part contains the end identifier of the RDB file and the checksum value of the entire file. This checksum value is used to check if the file has been tampered with after loading the RDB file into a Redis server.

The following diagram shows the components of an RDB file:

Now, let’s take a look at how the rdbSaveRio function generates each part of the RDB file. To help you understand the RDB file format and its contents, you can first follow these steps to prepare an RDB file.

Step 1: Start a Redis server for testing in the Redis directory on your computer by executing the following command:

./redis-server

Step 2: Execute the flushall command to clear the current database:

./redis-cli flushall

Step 3: Use redis-cli to log in to the Redis server you just started and execute the set command to insert a string-type key-value pair, and then execute the hmset command to insert a hash-type key-value pair. Execute the save command to save the current database content to an RDB file. The process is shown below:

127.0.0.1:6379> set hello redis
OK
127.0.0.1:6379> hmset userinfo uid 1 name zs age 32
OK
127.0.0.1:6379> save
OK

Now, you can find the generated RDB file in the directory where you executed the redis-cli commands. The file name should be dump.rdb.

However, because an RDB file is actually a binary data file, if you use a general text editing software such as Vim on Linux, you will see garbled characters when you open the RDB file. So here, I provide you with a tool. If you want to view the binary data and corresponding ASCII characters in the RDB file, you can use the od command on Linux. This command can display data in different number systems and display the corresponding ASCII characters.

For example, you can execute the following command to read the dump.rdb file and display its contents in hexadecimal, with each byte in the file displayed with its corresponding ASCII character:

od -A x -t x1c -v dump.rdb

The following code shows the part of the dump.rdb file starting from the file header after I used the od command to view the generated dump.rdb file. You can see that in the four lines of output, the first and third lines display the byte content of the dump.rdb file in hexadecimal. Each pair of hexadecimal numbers corresponds to one byte. The second and fourth lines display the ASCII characters corresponding to each byte generated by the od command.

This means that in the generated RDB file, if you want to convert it to ASCII characters, its file header actually already contains the REDIS string and some numbers, which is exactly the content included in the RDB file header.

Now, let’s see how the file header of an RDB file is generated.

Generating the File Header #

As mentioned earlier, the content of the RDB file header starts with a magic number, which records the version of the RDB file. In the rdbSaveRio function, the magic number is generated using the snprintf function, and its specific content is the string “REDIS” concatenated with the macro definition RDB_VERSION of the RDB version (defined in the rdb.h file, with a value of 9). Then, the rdbSaveRio function calls the rdbWriteRaw function (in the rdb.c file) to write the magic number to the RDB file, as shown below:

snprintf(magic, sizeof(magic), "REDIS%04d", RDB_VERSION); // Generate the magic number
if (rdbWriteRaw(rdb, magic, 9) == -1) goto werr; // Write the magic number to the RDB file

The rdbWriteRaw function we just used to write the magic number actually calls the rioWrite function (defined in the rdb.h file) to perform the write. The rioWrite function is the final write function for the RDB file content. It is responsible for writing the content in the write buffer to the RDB file based on the length of the data to be written. Here, you need to note that different functions are responsible for writing different parts of the content during the generation of the RDB file. However, these functions ultimately call the rioWrite function to perform the actual data write.

Okay, after writing the magic number in the RDB file header, the rdbSaveRio function then calls the rdbSaveInfoAuxFields function to write some attribute information related to the Redis server into the RDB file header, as shown below:

if (rdbSaveInfoAuxFields(rdb,flags,rsi) == -1) goto werr; // write attribute information

The rdbSaveInfoAuxFields function is implemented in the rdb.c file. It uses key-value pairs to record the attribute information of the Redis server in the RDB file header. The following table lists some of the main information recorded in the RDB file header, along with their corresponding keys and values. Take a look:

So, when the attribute value is a string, the rdbSaveInfoAuxFields function will call the rdbSaveAuxFieldStrStr function to write the attribute information. When the attribute value is an integer, the rdbSaveInfoAuxFields function will call the rdbSaveAuxFieldStrInt function to write the attribute information, as shown below:

if (rdbSaveAuxFieldStrStr(rdb,"redis-ver",REDIS_VERSION) == -1) return -1;
if (rdbSaveAuxFieldStrInt(rdb,"redis-bits",redis_bits) == -1) return -1;
if (rdbSaveAuxFieldStrInt(rdb,"ctime",time(NULL)) == -1) return -1;
if (rdbSaveAuxFieldStrInt(rdb,"used-mem",zmalloc_used_memory()) == -1) return -1;

Here, whether it is the rdbSaveAuxFieldStrStr function or the rdbSaveAuxFieldStrInt function, they will call the rdbSaveAuxField function to write the attribute value. The rdbSaveAuxField function is implemented in the rdb.c file and it completes the write of an attribute information in three steps.

In the first step, it calls the rdbSaveType function to write an opcode. The purpose of this opcode is to identify what the following content is in the RDB file. When writing attribute information, this opcode corresponds to the macro definition RDB_OPCODE_AUX (defined in the rdb.h file), with a value of 250 and a corresponding hexadecimal value of FA. This makes it easier for us to parse the RDB file. For example, when reading the RDB file, if the program reads the byte FA, it indicates that the following content is an attribute information.

Here, you need to note that the RDB file uses multiple opcodes to identify different content in the file. They are all defined in the rdb.h file. The following code shows some of the opcodes. Take a look:

#define RDB_OPCODE_IDLE       248   // Identifies LRU idle time
#define RDB_OPCODE_FREQ       249   // Identifies LFU access frequency information
#define RDB_OPCODE_AUX        250   // Identifies attribute information in RDB file header
#define RDB_OPCODE_EXPIRETIME_MS 252    // Identifies expiration time recorded in milliseconds
#define RDB_OPCODE_SELECTDB   254   // Identifies the database number to which subsequent key-value pairs belong in the file
#define RDB_OPCODE_EOF        255   // Identifies the end of the RDB file, used at the end of the file

In the second step, the rdbSaveAuxField function calls the rdbSaveRawString function (defined in the rdb.c file) to write the key of the attribute information. The key is usually a string. The rdbSaveRawString function is a general function used to write strings. It first records the length of the string and then records the actual string, as shown in the following figure. This length information allows the program to know how many bytes should be read for the current string based on it when parsing the RDB file.

However, in order to save space in the RDB file, if the actual content recorded in the string is an integer, the rdbSaveRawString function will also call the rdbTryIntegerEncoding function (defined in the rdb.c file) to try to encode the string using a compact structure. You can further read the rdbTryIntegerEncoding function for more details.

The following figure shows the basic execution logic of the rdbSaveRawString function. Take a look. It calls the rdbSaveLen function to write the string length and calls the rdbWriteRaw function to write the actual data.

In the third step, the rdbSaveAuxField function needs to write the value of the attribute information. Because the value of the attribute information is usually a string, similar to writing the key in the second step, the rdbSaveAuxField function calls the rdbSaveRawString function to write the value of the attribute information.

The following code shows the overall process of the rdbSaveAuxField function. Take another look:

ssize_t rdbSaveAuxField(rio *rdb, void *key, size_t keylen, void *val, size_t vallen) {
    ssize_t ret, len = 0;
    // Write the opcode
    if ((ret = rdbSaveType(rdb,RDB_OPCODE_AUX)) == -1) return -1;
    len += ret;
    // Write the key of the attribute information
    if ((ret = rdbSaveRawString(rdb,key,keylen)) == -1) return -1;
    len += ret;
    // Write the value of the attribute information
    if ((ret = rdbSaveRawString(rdb,val,vallen)) == -1) return -1;
    len += ret;
    return len;
}

Until now, the content of the RDB file header has been completed. I have drawn the part of the RDB file header that was just created in the diagram below, and marked the ASCII characters corresponding to the hexadecimal and some key information. You can refer to the legend to understand the code introduced earlier.

Next, the rdbSaveRio function will start writing the actual key-value pairs, which is the part of the file where the actual data is recorded. Let’s take a closer look.

Generating file data #

Because the key-value pairs on the Redis server may be stored in different databases, the rdbSaveRio function will perform a loop to iterate through each database and write the key-value pairs to the RDB file.

In this loop, the rdbSaveRio function will first write the SELECTDB opcode and the corresponding database number to the RDB file. This way, when the program parses the RDB file, it will know which database the next key-value pairs belong to. The process is shown below:

...
for (j = 0; j < server.dbnum; j++) { // loop through each database
...
// write SELECTDB opcode
if (rdbSaveType(rdb, RDB_OPCODE_SELECTDB) == -1) goto werr;
if (rdbSaveLen(rdb, j) == -1) goto werr;  // write the current database number j
...

The diagram below shows the information of the SELECTDB opcode in the RDB file I just created. As you can see, the database number is 0.

Next, the rdbSaveRio function will write the RESIZEDB opcode to indicate the record of the number of key-value pairs in the global hash table and the expired key hash table. Here is the code for this process:

...
db_size = dictSize(db->dict);   // get the size of the global hash table
expires_size = dictSize(db->expires);  // get the size of the expired key hash table
if (rdbSaveType(rdb, RDB_OPCODE_RESIZEDB) == -1) goto werr;  // write RESIZEDB opcode
if (rdbSaveLen(rdb, db_size) == -1) goto werr;  // write the size of the global hash table
if (rdbSaveLen(rdb, expires_size) == -1) goto werr;  // write the size of the expired key hash table
...

I have also drawn the content of the RESIZEDB opcode in the RDB file I just created in the diagram below. You can take a look.

As you can see, after the RESIZEDB opcode, the next record is the key-value pairs in the global hash table, with a count of 2, followed by the key-value pairs in the expired key hash table, with a count of 0. The information recorded in the RDB file matches our previous operation result because we only inserted two key-value pairs before generating the RDB file.

After recording this information, the rdbSaveRio function will continue with a loop. In this loop, the rdbSaveRio function will retrieve each key-value pair in the current database and use the rdbSaveKeyValuePair function (defined in rdb.c file) to write it to the RDB file. The basic loop process is as follows:

while ((de = dictNext(di)) != NULL) {
  sds keystr = dictGetKey(de);  // get the key of the key-value pair
  robj key, *o = dictGetVal(de);  // get the value of the key-value pair
  initStaticStringObject(key, keystr);  // create a String object for the key
  expire = getExpire(db, &key);  // get the expiration time of the key-value pair
  // write the key and value to the RDB file
  if (rdbSaveKeyValuePair(rdb, &key, o, expire) == -1) goto werr;
  ...
}

In this code, the rdbSaveKeyValuePair function is mainly responsible for actually writing the key-value pair to the RDB file. It will first write the expiration time, LRU idle time, or LFU access frequency of the key-value pair to the RDB file. When writing these information, the rdbSaveKeyValuePair function will first call the rdbSaveType function to write the opcode that identifies these information. You can see the code below.

if (expiretime != -1) {
  // write the opcode that identifies the expiration time
  if (rdbSaveType(rdb, RDB_OPCODE_EXPIRETIME_MS) == -1) return -1;

Hope this helps!

if (rdbSaveMillisecondTime(rdb,expiretime) == -1) return -1;
}
if (savelru) {
   ...
   //Write LRU idle time opcode indicator
   if (rdbSaveType(rdb,RDB_OPCODE_IDLE) == -1) return -1;
   if (rdbSaveLen(rdb,idletime) == -1) return -1;
}
if (savelfu) {
   ...
   //Write LFU access frequency opcode indicator
   if (rdbSaveType(rdb,RDB_OPCODE_FREQ) == -1) return -1;
   if (rdbWriteRaw(rdb,buf,1) == -1) return -1;
}


Okay, now rdbSaveKeyValuePair function is about to start writing the actual key-value pairs. In order to facilitate the recovery of key-value pairs when parsing RDB files, the rdbSaveKeyValuePair function will first call the rdbSaveObjectType function to write the type identifier of the key-value pair. Then it will call the rdbSaveStringObject to write the key of the key-value pair. Finally, it will call the rdbSaveObject function to write the value of the key-value pair. The process is as follows. These functions are all implemented in the rdb.c file:

if (rdbSaveObjectType(rdb,val) == -1) return -1;  //Write the type identifier of the key-value pair
if (rdbSaveStringObject(rdb,key) == -1) return -1; //Write the key of the key-value pair
if (rdbSaveObject(rdb,val,key) == -1) return -1; //Write the value of the key-value pair

Here, you need to note that the rdbSaveObjectType function determines the type identifier of the key-value pair written to the RDB based on the value type of the key-value pair. These type identifiers are macro-defined in the rdb.h file. For example, before I created the RDB file just now, the key-value pairs I wrote were of String type and Hash type. The Hash type uses a ziplist data structure to store because it has few elements by default. The values corresponding to these two type identifiers are as follows:

#define RDB_TYPE_STRING   0
#define RDB_TYPE_HASH_ZIPLIST  13

The contents of the record in the RDB file corresponding to the key-value pair "hello" "redis" that I just wrote are drawn in the figure below for you to see.

![](../images/786363a521ae6e3911c8c4002b53331d-20221013235857-6t98881.jpg)

As you can see, the type identifier at the beginning of this key-value pair is 0, which is consistent with the value defined by the RDB_TYPE_STRING macro. The key and value following it will first record the length information before recording the actual content.

Because the keys of the key-value pairs are all of String type, the rdbSaveKeyValuePair function uses the rdbSaveStringObject function to write them. The values of the key-value pairs have different types, so the rdbSaveObject function executes different code branches according to the type of the value, writing the content of the underlying data structure of the value to the RDB.

Okay, now we understand how the rdbSaveKeyValuePair function writes the key-value pairs into the RDB file. In this process, in addition to the key-value pair type, key and value, the expiration time, LRU idle time, or LFU access frequency of the key-value pair are also recorded in the RDB file. This generates the data section of the RDB file.

Finally, let's take a look at the generation of the file footer of the RDB file.

Generating the File Footer

After all the key-value pairs are written to the RDB file, the rdbSaveRio function can write the content of the file footer. The content of the file footer is relatively simple, mainly including two parts: the operation code flag indicating the end of the RDB file and the checksum value of the RDB file.

The rdbSaveRio function first calls the rdbSaveType function to write the end of the file operation code RDB_OPCODE_EOF, and then calls rioWrite to write the checksum value, as shown below:

...
//Write the end operation code
if (rdbSaveType(rdb,RDB_OPCODE_EOF) == -1) goto werr;

//Write the checksum value
cksum = rdb->cksum;
memrev64ifbe(&cksum);
if (rioWrite(rdb,&cksum,8) == 0) goto werr;
...

The figure below shows the file footer of the RDB file I just generated, you can take a look.

![](../images/158b7b6f7315e6376428f9cc3e04a515-20221013235857-3mq0vdd.jpg)

In this way, we have a comprehensive understanding of the entire process of generating the RDB file from the file header, the data section, to the file footer.

Summary #

In today’s lesson, I introduced the generation of Redis memory snapshot files, RDB. You should know that there are three entry functions for creating RDB files: rdbSave, rdbSaveBackground, and rdbSaveToSlavesSockets. These functions are called in the Redis source code, which determines when the RDB file generation is triggered.

In addition, you should also pay attention to the basic components of the RDB file and combine them with the execution process of the rdbSaveRio function to understand the generation of the RDB file header, file data section, and file footer. I have summarized the following two points to help you have a holistic understanding of the structure and content of the RDB file:

  • The RDB file uses multiple opcodes to identify different attribute information in Redis and uses type codes to identify different value types.
  • The content of the RDB file is self-contained, which means that whether it is attribute information or key-value pairs, the RDB file will be recorded according to the format of type, length, and actual data. This makes it easy for programs to parse the RDB file.

Finally, I would like to mention again that the RDB file contains all key-value pairs in the Redis database at a certain moment, as well as information such as their types, sizes, and expiration times. Once you understand the format and generation methods of the RDB file, you can develop programs to parse or load RDB files according to your needs.

For example, you can search for key-value pairs that consume a lot of memory in the RDB file, known as bigkeys, which are often needed when optimizing Redis performance. You can also analyze the distribution of the number and space usage of different types of key-value pairs to understand the characteristics of the business data. You can also load RDB files for testing or troubleshooting.

Of course, here’s a little tip for you: before you actually develop an RDB file analysis tool, you can take a look at redis-rdb-tools. It can help you analyze the content of RDB files. And if it doesn’t meet your customized requirements, you can use the knowledge learned in this lesson to develop your own RDB analysis tool.

One question per lesson #

Can you determine how many times the rdbSaveBackground function will be called and executed within the serverCron function? Also, can you explain the corresponding scenarios for each call?