Extra Meal 05 Useful Operational Tools for Redis

Extra Meal 05 Useful Operational Tools for Redis #

In today’s extra meal, I will share with you some useful operation and maintenance tools for Redis.

When using Redis in application, we often face operational tasks such as monitoring the running status of Redis, data migration, deployment, and operation of master-slave clusters and sharded clusters. Next, I will introduce you to some tools from these three aspects. Let’s start by learning about tools that monitor the real-time operation status of Redis. These tools all use a monitoring command provided by Redis: INFO.

The Basic Monitoring Command: INFO Command #

The INFO command provided by Redis itself returns rich instance monitoring information and serves as the foundation for Redis monitoring tools.

When using the INFO command, you can include a parameter called “section”. There are several values for this parameter, and accordingly, the INFO command will return different types of monitoring information. I have divided the return information of the INFO command into 5 major categories, and some categories also contain different monitoring contents, as shown in the following table:

When monitoring the running status of Redis, the result returned by the INFO command is very useful. If you want to understand the detailed meanings of all the parameter return results of the INFO command, you can refer to the official website of Redis for an introduction. Here, I will provide you with several parameters that you need to focus on when operating and maintaining, as well as their important return results.

First of all, whether you are running a single instance or a cluster, I recommend focusing on the return results of the stat, commandstat, cpu, and memory parameters. These parameters include the execution status of commands (such as the number of command executions and execution time, and the CPU resources used by the commands), the memory usage (such as the amount of memory used and the memory fragmentation rate), and the CPU resource usage. This can help us judge the running status and resource consumption of the instances.

In addition, if you enable the RDB or AOF feature, you need to pay attention to the return results of the persistence parameter, which allows you to view the execution status of RDB or AOF.

If you are using a master-slave cluster, you should focus on the return results of the replication parameter, which contains real-time status updates of the master-slave synchronization.

However, the INFO command only provides text-based monitoring results without visualization. Therefore, in practical applications, we can also use some third-party open-source tools to visualize the return results of the INFO command. Next, I will talk about Prometheus, which can visualize the statistical results of Redis through plugins.

Monitoring Redis with Prometheus Redis-exporter #

Prometheus is an open-source system monitoring and alerting framework. Its core functionality is to pull monitoring data from the systems being monitored and visualize it using the Grafana tool. The monitoring data can also be stored in a time series database for historical querying by operators. Additionally, Prometheus can detect if the monitored metrics exceed predefined thresholds and trigger alerts.

These functionalities are crucial for day-to-day operations management of systems. Prometheus provides a framework for using these functionalities, and as long as we can obtain monitoring data from the systems being monitored, we can use Prometheus for operations monitoring.

Prometheus offers plugin functionality to monitor a system, and we refer to these plugins as exporters. Each exporter is essentially a component that collects monitoring data. The data collected by an exporter conforms to Prometheus’ requirements, and once Prometheus obtains this data, it can be visualized and stored.

Redis-exporter is an exporter specifically built for monitoring Redis. It provides Prometheus with runtime status and various statistics obtained from monitoring the Redis INFO command, enabling visualization and alerting. Currently, Redis-exporter supports Redis versions 2.0 to 6.0, making it applicable to a wide range of setups.

In addition to obtaining the runtime status of Redis instances, Redis-exporter can also monitor the size of key-value pairs and the number of elements in collection type data. This can be achieved by running Redis-exporter with the check-keys command-line option.

Furthermore, we can develop a Lua script to customize the collection of required monitoring data. We can then use the scripts command-line option to make Redis-exporter execute this specific script, thus meeting the diverse monitoring requirements of the business layer.

Finally, I would like to share two more tools with you: redis-stat and Redis Live. These two tools, unlike Redis-exporter, are lightweight monitoring tools. They are respectively developed in Ruby and Python and visualize the runtime status information provided by the INFO command. Although these two tools have seen fewer updates lately, they serve as good references if you want to develop your own Redis monitoring tool.

In addition to monitoring the runtime status of Redis, another common operations task is data migration. Next, let’s learn about tools for data migration.

Data Migration Tool Redis-shake #

Sometimes, we need to migrate data between different instances. Currently, a commonly used data migration tool is Redis-shake, which is developed by Alibaba Cloud Redis and MongoDB teams for Redis data synchronization.

The basic principle of Redis-shake is to start the Redis-shake process, which simulates a Redis instance. Then, the Redis-shake process performs a full-data synchronization with the source instance from which the data is migrated.

This process is similar to the full-data synchronization of Redis master-slave instances.

The source instance acts as the master database, while Redis-shake acts as the slave database. First, the source instance transfers the RDB file to Redis-shake. Redis-shake then sends the RDB file to the destination instance. Afterwards, the source instance sends incremental commands to Redis-shake, and Redis-shake is responsible for synchronizing these incremental commands to the destination instance.

The following diagram shows the process of data migration using Redis-shake:

Redis-shake has a major advantage in supporting various types of migrations.

First, it supports data migration between single instances as well as between clusters.

Additionally, some Redis sharded clusters (such as Codis) use a proxy to receive request operations, and Redis-shake also supports data migration with a proxy.

Furthermore, because Redis-shake is developed by the Alibaba Cloud team, it not only supports open-source Redis versions, but also supports migrating Redis instances both on-premises and on the cloud, helping us achieve the goal of migrating Redis services to the cloud.

After data migration, we usually need to compare the data consistency between the source and destination instances. If there are inconsistent data, we need to identify them, remove them from the destination instance, or migrate the inconsistent data again.

Here, I would like to introduce another tool for data consistency comparison, which is developed by the Alibaba Cloud team, called Redis-full-check.

The working principle of Redis-full-check is simple: it compares the data in the source and destination instances in order to complete data verification. However, in order to reduce the comparison overhead, Redis-full-check adopts a multi-round comparison method.

In the first round of verification, Redis-full-check finds all the keys in the source instance, then retrieves the corresponding values from both the source and destination instances for comparison. After the first round of comparison, Redis-full-check records the inconsistent data in the destination instance compared to the source instance in a SQLite database.

Starting from the second round of verification, Redis-full-check only compares the inconsistent data recorded in the database from the previous round.

To avoid affecting the normal request processing of instances, Redis-full-check pauses for a period of time after each round of comparison. As the incremental synchronization progresses in Redis-shake, the inconsistent data in the source and destination instances will gradually decrease, so we do not need to perform many rounds of verification.

We can set the number of comparison rounds ourselves. Specifically, when running the redis-full-check command, we set the value of the comparetimes parameter to the desired number of rounds to compare.

After all rounds of comparison are completed, the data recorded in the database represents the final difference between the source and destination instances.

There is one thing to note here. Redis-full-check provides three comparison modes, which can be set using the comparemode parameter. The comparemode parameter can take three values with the following meanings:

KeyOutline: only compares if the key values are equal.
ValueOutline: only compares if the length of the value is equal.
FullValue: compares if the key value, value length, and value itself are equal.

When using Redis-full-check, we can choose the appropriate comparison mode based on the required level of data consistency for our business. If high consistency is required, we set the comparemode parameter to FullValue.

Alright, finally, I would like to introduce CacheCloud, a tool for Redis cluster operations and management.

Cluster Management Tool CacheCloud #

CacheCloud is a cloud platform developed by Sohu for Redis operations and management. It implements automatic deployment and management of master-slave clusters, sentinel clusters, and Redis Cluster. Users can perform operations directly on the platform’s management interface.

In response to common cluster operations and maintenance needs, CacheCloud provides 5 operational actions.

Offline instance: Shut down the instance and related monitoring tasks.
Online instance: Restart the offline instance and monitor it.
Add slave node: Add a slave node to the master-slave cluster.
Fault switching: Manually complete the failover of Redis Cluster master and slave nodes.
Configuration management: After the user submits a configuration modification work order, the administrator reviews and completes the configuration modification.

Of course, as an operations management platform, CacheCloud not only provides operational actions, but also provides rich monitoring information.

CacheCloud not only collects real-time running status information provided by the INFO command for instances and visualizes it, but also saves the running status information of instances, such as memory usage, client connections, and key-value data volume. In this way, when problems occur with Redis running, operations personnel can query the saved historical records and analyze them in combination with the running status information at that time.

If you want a unified platform to centralize the management tasks related to Redis instances, CacheCloud is a good tool.

Summary #

In this lesson, I introduced you to several Redis operation and maintenance tools.

First, we learned about the INFO command in Redis, which is the foundation of monitoring tools. Monitoring tools will use the information provided by the INFO command for further processing. We also learned about three operation and maintenance tools used to monitor the real-time running status of Redis: Redis-exporter, redis-stat, and Redis Live.

Regarding data migration, we can use the Redis-shake tool or migrate using RDB or AOF files.

When operating and maintaining Redis, the various open-source tools mentioned earlier can meet many of our needs. However, sometimes different business lines may have different requirements for Redis operation and maintenance. Using ready-made open-source tools may not satisfy all the requirements. In such cases, I recommend that you consider customizing or developing your own tools based on the existing open-source tools to better meet the specific needs of your business.

Question of the Lesson #

As usual, I have a small question for you: Have you used any good DevOps tools in practical applications?

Feel free to write down your thoughts and answers in the comment section. Let’s communicate and discuss together. If you find today’s content helpful, you’re also welcome to share it with your friends or colleagues. See you in the next lesson.