17 RocketMQ Cluster Performance Optimization #

Introduction #

In this article, we will optimize the RocketMQ cluster from two dimensions: system parameters and cluster parameters. The goal is to make the RocketMQ run more smoothly. Stability is often more important than simply improving TPS (transactions per second). Based on the actual production environment, this article provides examples of pitfalls caused by parameter settings that led to unstable clusters and affected business operations.

System Parameter Tuning #

After extracting the RocketMQ installation package, there is a file called os.sh in the bin directory. This file provides system parameter configurations recommended by RocketMQ official. Usually, these parameters can meet the system requirements, but they can also be adjusted according to the actual situation. It is important to note that Linux kernel version 2.6 or below should not be used. It is recommended to use Linux kernel version 3.10 or above. If you are using CentOS, you can choose CentOS 7 or above. Problems that occur when using Linux kernel version 2.6 will be discussed in the later section on pitfalls.

Maximum File Count #

Set the maximum number of files opened by the user:

vim /etc/security/limits.conf
# End of file
baseuser soft nofile 655360
baseuser hard nofile 655360
* soft nofile 655360
* hard nofile 655360

System Parameter Settings #

The adjustment of system parameters follows the recommendations from the official documentation. Below, we explain each parameter. You can directly execute sh os.sh to complete the system parameter configuration, or you can manually add the following content to the vim /etc/sysctl.conf file and then execute sysctl -p to make it take effect.

vm.overcommit_memory=1
vm.drop_caches=1
vm.zone_reclaim_mode=0
vm.max_map_count=655360
vm.dirty_background_ratio=50
vm.dirty_ratio=50
vm.dirty_writeback_centisecs=360000
vm.page-cluster=3
vm.swappiness=1

Parameter explanations:

Cluster Parameter Tuning #

Production Environment Configuration #

Here is a configuration file used in production environments, along with an explanation of its parameters. Just make slight modifications to the cluster name and you can use it as the configuration file for your production environment.

Example configuration:

brokerClusterName=testClusterA
brokerName=broker-a
brokerId=0
listenPort=10911
namesrvAddr=x.x.x.x:9876;x.x.x.x::9876
defaultTopicQueueNums=16
autoCreateTopicEnable=false
autoCreateSubscriptionGroup=false
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88
storePathRootDir=/data/rocketmq/store
storePathCommitLog=/data/rocketmq/store/commitlog
storePathConsumeQueue=/data/rocketmq/store/consumequeue
storePathIndex=/data/rocketmq/store/index
storeCheckpoint=/data/rocketmq/store/checkpoint
abortFile=/data/rocketmq/store/abort
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
maxTransferCountOnMessageInMemory=1000
transientStorePoolEnable=false
warmMapedFileEnable=false
pullMessageThreadPoolNums=128
slaveReadEnable=true
transferMsgByHeap=true
waitTimeMillsInSendQueue=1000

Parameter explanations:

Optimization Recommendations #

There are several properties of the broker that may affect the stability of the cluster’s performance. They are explained below.

1. Enable asynchronous flushing

Except for certain payment scenarios or low TPS scenarios (e.g., TPS below 2000), it is recommended to enable asynchronous flushing to improve cluster throughput.

flushDiskType=ASYNC_FLUSH

2. Enable slave read permission

The size of messages occupying physical memory is configured by accessMessageInMemoryMaxRatio, with a default value of 40%. If the consumed messages are not in memory, enabling slaveReadEnable will read the messages from the slave node, thereby increasing the memory utilization of the master.

slaveReadEnable=true

3. Number of messages pulled in one request

The number of messages pulled in one request during consumption is determined jointly by the broker and the consumer client, with a default value of 32. The broker-side parameter is set by maxTransferCountOnMessageInMemory, and the consumer-side parameter is set by pullBatchSize. It is recommended to set a larger value on the broker side, such as 1000, to leave more room for adjustment on the consumer side.

maxTransferCountOnMessageInMemory=1000

4. Waiting time in the sending queue

When a message is sent to the broker, the waiting time in the queue is set by the waitTimeMillsInSendQueue parameter, with a default value of 200ms. It is recommended to set a larger value, such as 1000ms to 5000ms. If set too short, the sending client will experience timeouts.

waitTimeMillsInSendQueue=1000

5. Master-Slave asynchronous replication

To improve cluster performance, in production environments, it is recommended to set the role to ASYNC_MASTER for master-slave asynchronous replication, as the performance of master-slave synchronous replication is relatively low in stress testing.

brokerRole=ASYNC_MASTER

6. Increase cluster stability

To improve cluster stability, the following three parameters are particularly important. They will also be mentioned in the troubleshooting cases.

Disable off-heap memory:

transientStorePoolEnable=false

Disable file warming:

warmMapedFileEnable=false

Enable heap memory transfer:

transferMsgByHeap=true