04 Redis Persistence Aof

04 Redis Persistence AOF #

There is a risk of data loss when using RDB persistence. This is because RDB persistence has a certain time interval, and if the Redis service terminates unexpectedly during this time period, the latest data will be lost.

Conditions that may cause the Redis service to terminate unexpectedly include:

The machine where Redis is installed stops running, such as a blue screen or system crash.
The machine where Redis is installed experiences a power failure, such as a sudden power loss.
Using commands like kill -9 Redis_PID.

So how do we solve these problems? Redis provides us with another persistence solution called AOF.

1 Introduction #

AOF (Append Only File) means that Redis can record every key-value operation to a file (appendonly.aof).

2 Persistence query and configuration #

1) Query AOF status #

Use the config get appendonly command, as shown in the following image:

The first line represents the name of the AOF file, and the last line indicates the status of AOF. “yes” indicates that it is enabled, while “no” indicates that it is not enabled.

2) Enable AOF persistence #

AOF persistence is disabled by default in Redis. To enable AOF persistence, there are two ways:

Through the command line
By modifying the configuration file (redis.conf)

Let’s look at how to implement these two methods separately.

① Enable AOF via command line #

To enable AOF persistence via the command line, use the config set appendonly yes command, as shown in the following image:

Advantages and disadvantages of enabling AOF via the command line: The advantage of enabling AOF via the command line is that it does not require restarting the Redis service. The disadvantage is that if the Redis service is restarted, the configuration set via the command line will be lost.

② Enable AOF via configuration file #

The Redis configuration file is named redis.conf and it is located in the root directory of Redis. You can use the command config get dir to get the root directory of Redis, as shown in the following image:

Just set appendonly yes in the configuration file. By default, appendonly no means that AOF persistence is disabled.

Advantages and disadvantages of enabling AOF via the configuration file: The disadvantage of modifying the configuration file is that every time you make changes, you need to restart the Redis service for the changes to take effect. The advantage is that the configuration settings in the configuration file will not be lost regardless of how many times the Redis service is restarted.

3 Triggering persistence #

After AOF persistence is enabled, it will be triggered as long as certain conditions are met. AOF triggering conditions can be divided into automatic triggering and manual triggering.

1) Automatic triggering #

There are two cases where AOF persistence is automatically triggered: Triggered by the AOF strategy and triggered by AOF rewriting. The latter will be explained in detail in the latter part of this article. Here, we focus on the AOF persistence strategies.

There are three AOF persistence strategies:

everysec: The AOF file is synchronized to disk every second.
always: The AOF file is synchronized to disk after every write command.
no: Data is not synchronized to disk.
- always: Each Redis operation command is written to disk, with at most one piece of data being lost.
- everysec: Sync to disk every second, with at most one second of data being lost.
- no: No rules for writing to disk are set. The operating system determines when to write to disk. Linux defaults to writing data to disk every 30 seconds.

These three configurations can be set in the Redis configuration file (redis.conf) as shown in the code below:

# Enable the strategy of writing to disk every second
appendfsync everysec

Tip: Since writing to disk each time will have a certain impact on the performance of Redis, it is recommended to set the appropriate strategy according to the actual situation of the user. Generally, setting the frequency of writing to disk every second will meet the requirements of most scenarios.

There are two situations that trigger automatic persistence, as shown in the following image:

2) Triggering Manually #

Executing the bgrewriteaof command in the client can trigger AOF persistence manually, as shown in the image below:

After executing the bgrewriteaof command, AOF persistence will be triggered.

4 AOF File Rewriting #

AOF persists (saves) data by recording the execution commands of Redis over time. As a result, as time goes on, the size of the AOF file increases, which not only increases the storage pressure on the server but also slows down the speed of Redis restart. To solve this problem, Redis provides AOF file rewriting.

1) What is AOF File Rewriting? #

AOF file rewriting directly reads the current state of the Redis server and compresses it to save it as an AOF file. For example, if a counter is increased and modified 99 times, if AOF file rewriting is not performed, there will be 100 records of command execution information in the persistence file. However, after AOF file rewriting, only one record of the final result of the counter will be recorded, removing all the invalid information.

2) Implementation of AOF File Rewriting #

To trigger AOF file rewriting, two conditions must be met. These two conditions are also configured in the Redis configuration file:

auto-aof-rewrite-min-size: The minimum file size for AOF rewriting is allowed, with a default value of 64MB.
auto-aof-rewrite-percentage: The size ratio for AOF file rewriting, with a default value of 100, which means 100%. That is, only when the current AOF file is more than twice the size of the last AOF file, will AOF file rewriting be triggered.

To query the values of auto-aof-rewrite-min-size and auto-aof-rewrite-percentage, you can use the config get xxx command, as shown in the following image:

Tip: AOF file rewriting will only be triggered if both the conditions set by auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are met.

Note: The bgrewriteaof command can be used to automatically trigger AOF file rewriting.

3) AOF File Rewriting Process #

AOF file rewriting generates a completely new file and saves the least number of operation commands for the current data to the new file. After all the data is saved to the new file, Redis swaps the two files and appends the latest persistent operation commands to the new file.

5 Configuration Explanation #

Reasonable configuration of AOF can ensure the efficient and stable operation of Redis. The following is a complete list of AOF configuration information and explanations. The configuration parameters for AOF are located in the Redis configuration file, which is the redis.conf file in the Redis root directory. The configuration parameters and their descriptions are as follows:

# Enable AOF, default is disabled
appendonly no

# AOF default file name
appendfilename "appendonly.aof"

# AOF persistence strategy configuration
# appendfsync always
appendfsync everysec
# appendfsync no

# AOF file rewrite percentage, default value is 100, which means 100%. 
# Only when the current AOF file is twice as large as the last AOF file, AOF file rewrite will be triggered.
auto-aof-rewrite-percentage 100

# Minimum file capacity allowed for AOF rewrite
auto-aof-rewrite-min-size 64mb

# Enable loading AOF file validation at startup, default value is yes.
# If set to no, Redis will not start until the AOF file is manually repaired.
aof-load-truncated yes

One of the important parameters is appendfsync, which is used to set the persistence strategy for AOF. You can choose to store the AOF file based on time intervals or number of operations. The three possible values for this parameter are explained at the beginning of the article and will not be repeated here.

6 Data Recovery #

1) Normal Data Recovery #

In normal cases, as long as AOF persistence is enabled and a valid appendonly.aof file is provided, Redis will automatically load the AOF file and start when starting Redis, as shown in the following figure:

The message DB loaded from append only file...... indicates that Redis has loaded the AOF persistence file when starting the Redis server.

Tip: By default, the appendonly.aof file is saved in the Redis root directory.

Persistence File Loading Rules

If only AOF persistence is enabled, Redis will only load the AOF file (appendonly.aof) when starting, to perform data recovery.
If only RDB persistence is enabled, Redis will only load the RDB file (dump.rdb) when starting, to perform data recovery.
If both RDB and AOF persistence are enabled, Redis will only load the AOF file (appendonly.aof) when starting, to perform data recovery.

In the case of AOF being enabled, even if the AOF file does not exist but there is only an RDB file, Redis will not load the RDB file. The loading process for AOF and RDB is shown in the following figure:

2) Simple Abnormal Data Recovery #

If the server crashes during AOF write or the AOF storage becomes full, the last command in the AOF file may be truncated, resulting in an abnormal AOF file.

In the case of an abnormal AOF file, if the Redis configuration is modified to set aof-load-truncated to yes, Redis will ignore the last command and start successfully. The result will be as follows:

* Reading RDB preamble from AOF file...
* Reading the remaining AOF tail...
# !!! Warning: short read while loading the AOF file !!!
# !!! Truncating the AOF at offset 439 !!!
# AOF loaded anyway because aof-load-truncated is enabled

3) Complex Abnormal Data Recovery #

AOF files can have even worse situations where they are not only truncated but also have corrupted commands in the middle. In this case, Redis will display an error message and abort when trying to start, the error message is as follows:

* Reading the remaining AOF tail...
# Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>

The solution for this type of problem is as follows:

First, use the AOF repair tool to detect the problem. Enter the redis-check-aof command in the command line, and it will jump to the command line where the problem occurred. At this point, you can try to manually repair the file.
If manual repair is not possible, you can use redis-check-aof --fix to automatically repair the abnormal AOF file. However, executing this command may result in all data from the exception to the end of the file being discarded.

7 Pros and Cons #

AOF Pros #

AOF persistence provides more complete data. AOF offers three saving strategies: every operation saved, saved every second, or following the system’s persistence strategy. Saving every second is a good choice considering both data security and performance. It is also the default strategy. Even in the event of an accident, at most only 1 second of data will be lost.
AOF uses the append-only writing method, so file corruption will not be an issue. Even if for some unexpected reason, the last operation’s persisted data was only halfway written, it can be easily repaired using the redis-check-aof tool.
AOF persistence files are easy to understand and parse. It stores all Redis key-value operation commands in a file on disk. Even if you accidentally use the flushall command to delete all key-value information, you can just remove the last flushall command from the AOF file and restart Redis to recover the mistakenly deleted data.

AOF Cons #

For the same data set, the AOF file is larger than the RDB file.
In cases when Redis is under high load, RDB performs better than AOF.
RDB persists the entire Redis dataset in the form of a snapshot, whereas AOF only appends every executed command to the AOF file. Therefore, in theory, RDB is more robust than AOF.

8 Conclusion #

AOF provides more complete data storage as it records every key-value change in Redis. It allows you to choose to save data on every operation or to save it every second. AOF persistence files are easy to read. However, compared to the binary RDB files, they occupy more storage space. To solve this problem, AOF provides an automatic rewriting mechanism to minimize the space occupied by AOF files. AOF also provides convenient commands for recovering from abnormal files, such as redis-check-aof --fix, which ensures the usage of AOF is reliable.

References & Acknowledgements: