19 Redis Pipeline Technology #

Pipeline technology is a client-side batch processing technique provided by Redis that allows multiple Redis commands to be processed at once, thereby improving the performance of the entire interaction.

By default, Redis executes commands in a single line. The client sends a request to the server, the server receives and processes the request, and then returns the results to the client. This processing mode works fine when there are no frequent requests.

However, when there are a large number of concentrated requests, it can lead to waste of network resources because each request has to go through the process of sending request and receiving response. In such cases, pipeline technology is needed to aggregate all the commands and send them to the server at once, and then respond to the client all at once. This greatly improves the response speed of Redis.

In the normal command mode, as shown in the figure below:

In the pipeline mode, as shown in the figure below:

Tip: The more commands there are in the pipeline, the more significant the role of the pipeline technology is, and the higher the execution efficiency compared to the normal mode.

What problem does pipeline technology solve? #

Pipeline technology solves the problem of waste of network resources when multiple commands are requested at once. It speeds up the response of Redis and allows Redis to operate at a higher speed. However, it should be noted that pipeline technology is essentially a feature provided by the client, not the Redis server itself.

Using pipeline technology #

In this article, we use the Pipeline object provided by the Jedis client to implement pipeline technology. First, obtain the Pipeline object, then set the commands that need to be executed for the Pipeline object, and finally use the sync() method or the syncAndReturnAll() method to execute these commands in a unified way. The code is as follows:

public class PipelineExample {
    public static void main(String[] args) {
        Jedis jedis = new Jedis("127.0.0.1", 6379);
        // Record the start time of execution
        long beginTime = System.currentTimeMillis();
        // Get the Pipeline object
        Pipeline pipe = jedis.pipelined();
        // Set multiple Redis commands
        for (int i = 0; i < 100; i++) {
            pipe.set("key" + i, "val" + i);
            pipe.del("key"+i);
        }
        // Execute the commands
        pipe.sync();
        // Record the end time of execution
        long endTime = System.currentTimeMillis();
        System.out.println("Execution time: " + (endTime - beginTime) + " milliseconds");
    }
}

The result of the above program execution is as follows:

Execution time: 297 milliseconds

If you want to receive the execution results of all commands in the pipeline, you can use the syncAndReturnAll() method. The sample code is as follows:

public class PipelineExample {
    public static void main(String[] args) {
        Jedis jedis = new Jedis("127.0.0.1", 6379);
        // Get the Pipeline object
        Pipeline pipe = jedis.pipelined();
        // Set multiple Redis commands
        for (int i = 0; i < 100; i++) {
            pipe.set("key" + i, "val" + i);
        }
        // Execute the commands and return the results
        List<Object> res = pipe.syncAndReturnAll();
        for (Object obj : res) {
            // Print the results
            System.out.println(obj);
        }
    }
}

Pipeline technology vs normal commands #

The time it takes to execute a for loop using pipeline technology in the above example is 297 milliseconds. Next, let’s execute this loop with normal commands and see the execution time of the program. The code is as follows:

public class PipelineExample {
    public static void main(String[] args) {
        Jedis jedis = new Jedis("127.0.0.1", 6379);
        // Record the start time of execution
        long beginTime = System.currentTimeMillis();
        for (int i = 0; i < 100; i++) {
            jedis.set("key" + i, "val" + i);
            jedis.del("key"+i);
        }
        // Record the end time of execution
        long endTime = System.currentTimeMillis();
        System.out.println("Execution time: " + (endTime - beginTime) + " milliseconds");
    }
}

The result of the above program execution is as follows:

Execution time: 17276 milliseconds

Conclusion

From the above result, it can be seen that the execution time of the pipeline is 297 milliseconds, while the execution time of normal commands is 17276 milliseconds. The pipeline technology is 58 times faster than normal execution.

Things to note about pipeline technology #

Although pipeline technology has its advantages, there are a few details to note when using it:

• The number of commands sent is not limited, but the maximum storage size of the input buffer, which is the command, is 1GB. When the sent commands exceed this limit, the commands will not be executed and the Redis server will disconnect the connection;
• If the data in the pipeline is too large, it may cause the client to wait for a long time, resulting in network blocking;
• Some clients also have their own settings for the buffer size. If the pipeline commands have not been executed or have not been executed completely, this situation can be investigated or reduce the number of commands in the pipeline and try to execute them again.

Summary #

Using pipeline technology can solve the problem of network waiting when multiple commands are executed. It aggregates multiple commands and sends them to the server to be processed in one go, avoiding the need to wait for each command to be executed. This effectively improves the execution efficiency of the program. However, when using pipeline technology, attention should be paid to avoid sending too many commands or too much data in the pipeline, which may cause network blocking.