05 Redis Hybrid Persistence

05 Redis Hybrid Persistence #

RDB and AOF persistence have their own advantages and disadvantages. RDB may result in data loss within a certain period of time, while AOF, due to its large file size, may affect the startup speed of Redis. In order to leverage the benefits of both RDB and AOF, Redis 4.0 introduced a new mixed persistence approach.

With mixed persistence enabled, when AOF is being rewritten, Redis’s persistence data is written in RDB format at the beginning of the AOF file, followed by the data being appended to the end of the file in AOF format.

The data storage structure of mixed persistence is shown in the following diagram: image.png

1 Enabling mixed persistence #

To check if mixed persistence is enabled, you can use the config get aof-use-rdb-preamble command. The result would be as shown in the following diagram: image.png In the result, “yes” indicates that mixed persistence is enabled, while “no” indicates that it is disabled. The default value in Redis 5.0 is “yes”. For other versions of Redis, you first need to check if mixed persistence is already enabled. If it is disabled, you can enable it using either of the following two methods:

  • Enable via command line
  • Enable by modifying the Redis configuration file

1) Enable via command line #

Using the command config set aof-use-rdb-preamble yes, the result would be as shown in the following diagram: image.png

Tip: The disadvantage of setting a configuration via the command line is that the configuration will become invalid after restarting the Redis service.

2) Enable by modifying the Redis configuration file #

In the root directory of Redis, find the redis.conf file and change the aof-use-rdb-preamble no configuration in the file to aof-use-rdb-preamble yes, as shown in the following diagram: image.png

2 Running the example #

When mixed persistence is disabled and you use bgrewriteaof to trigger AOF file rewriting, you can check the persistence log of the appendonly.aof file, as shown in the following diagram: image.png As you can see, when mixed persistence is disabled, the AOF persistence file stores the file in a standard AOF format. When mixed persistence mode is enabled and you use the bgrewriteaof command to trigger AOF file rewriting, the content of the appendonly.aof file is as shown in the following diagram: image.png It can be seen that the content stored in the appendonly.aof file is in RDB format that starts with REDIS, rather than in AOF format.

3 Data recovery and source code analysis #

Data recovery in mixed persistence is the same as in AOF persistence. You just need to place the appendonly.aof file in the root directory of Redis, and when Redis starts up with AOF persistence enabled, it will automatically load and restore the data. The startup information of Redis is shown in the following diagram: image.png As you can see, Redis loads the content of the AOF file during server initialization.

1) Loading process of mixed persistence #

The loading process of mixed persistence is as follows:

  1. Check if AOF persistence is enabled. If enabled, continue to the next step. If disabled, execute the process of loading the RDB file.
  2. Check if the appendonly.aof file exists. If the file exists, continue to the next step.
  3. Check if the beginning of the AOF file is in RDB format. If so, load the RDB content first, then load the remaining AOF content.
  4. If the beginning of the AOF file is not in RDB format, load the entire file in AOF format.

The process of loading AOF is shown in the following diagram: image.png 2) Source code analysis

Redis determines whether the beginning of the AOF file is in RDB format by checking the keyword REDIS. The beginning of an RDB file always starts with the keyword REDIS. The code that performs this check can be found in src/aof.c of Redis, and the core code is as follows:

char sig[5]; /* "REDIS" */
if (fread(sig,1,5,fp) != 5 || memcmp(sig,"REDIS",5) != 0) {
    // AOF file is not in RDB format, not a mixed persistence file
    if (fseek(fp,0,SEEK_SET) == -1) goto readerr;
} else {
    /* RDB preamble. Pass loading the RDB functions. */
    rio rdb;

    serverLog(LL_NOTICE,"Reading RDB preamble from AOF file...");
    if (fseek(fp, 0, SEEK_SET) == -1) goto readerr;
    rioInitWithFile(&rdb, fp);
    // The beginning of the AOF file is in RDB format, so we load the RDB first and then the AOF
    if (rdbLoadRio(&rdb, NULL, 1) != C_OK) {
        serverLog(LL_WARNING, "Error reading the RDB preamble of the AOF file, AOF loading aborted");
        goto readerr;
    } else {
        serverLog(LL_NOTICE, "Reading the remaining AOF tail...");
    }
}
// Load data in AOF format

It can be seen that Redis determines whether a file is a mixed persistence file by checking whether the beginning of the AOF file is the REDIS keyword.

Tip: The beginning of the AOF format is *, while the beginning of the RDB format is REDIS.

4 Advantages and Disadvantages #

Advantages of mixed persistence:

  • Combined with the advantages of RDB and AOF persistence, the beginning is in the RDB format, which allows Redis to start faster. At the same time, it combines the advantages of AOF, reducing the risk of data loss.

Disadvantages of mixed persistence:

  • The AOF file has added content in the RDB format, making the AOF file difficult to read;
  • Poor compatibility. If mixed persistence is enabled, this mixed persistence AOF file cannot be used in Redis versions before 4.0.

5 Best Practices for Persistence #

Persistence guarantees the absence of data loss, but also slows down the running speed of Redis. How can we make the best use of Redis’s persistence function? The best practices for Redis persistence can be considered from the following aspects.

1) Control persistence switches #

Users can consider closing Redis persistence based on the actual business situation. If the loss of data is not sensitive, Redis persistence can be disabled so that all key-value operations are performed in memory, ensuring the highest efficiency of running Redis. To disable persistence:

  • Disable RDB persistence using the command: config set save ""
  • Disable AOF and mixed persistence using the command: config set appendonly no

2) Master-slave deployment #

Using master-slave deployment, one server is used to respond to the main business, and the other is used for data persistence. This allows Redis to run more efficiently.

3) Use mixed persistence #

Mixed persistence combines the advantages of RDB and AOF. It is enabled by default in Redis 5.0.

4) Use machines with higher configurations #

Redis does not require high CPU resources, but it does require high memory and disk resources because Redis mostly performs read and write operations. Using more memory and faster disks can greatly improve Redis performance.

References & Acknowledgements: https://redis.io/topics/persistence https://blog.csdn.net/qq_36318234/article/details/79994133 https://www.cnblogs.com/wdliu/p/9377278.html