25 Behind the Rocket Mq Nameserver Design Concepts

25 Behind the RocketMQ Nameserver - Design Concepts #

In the overall architecture of RocketMQ, Nameserver occupies a similar position as ZooKeeper in the ZooKeeper and Dubbo service-oriented architecture system, serving as the “registry center”. In RocketMQ, routing information mainly refers to the queue information of a topic, which means the distribution of queues for a topic among brokers.

Nameserver Working Mechanism #

1

The main participants in the registration and discovery of a topic are Nameserver, Producer, Consumer, and Broker. The interaction characteristics and connectivity are as follows:

  • Nameserver: The naming server, consisting of multiple machines in a cluster, with no communication between the machines.
  • Broker: The broker message server, sends heartbeat packets to each Nameserver in Nameserver every 30 seconds. Therefore, the routing information about the topic in Nameserver comes from the Broker. Due to this registration mechanism, and because the Nameservers are not interconnected, if there is a network partition or other factors, such as network interruption between broker-a and one of the Nameservers in the cluster, inconsistent data may occur between the two Nameservers. The specific impact will be discussed later in this document.
  • Producer, Consumer: Message sender and message consumer, connect to only one server in the Nameserver cluster at a time, and update the routing information of the topic every 30 seconds.

In addition, Nameserver scans the live status of brokers regularly. One of the criteria is that if there is no heartbeat information from a broker for 120 seconds continuously, all queue information about that broker is removed from the topic routing table. This ensures that message senders do not send messages to brokers that have failed, improving the high availability of message sending.

The registration center mode used by Nameserver is the PULL mode. The next section will provide a detailed introduction to the implementation approaches of the mainstream registration center, so as to explain the architectural selection strategies.

Two Approaches for Designing Registration Centers #

PUSH Mode #

When it comes to service registration centers, everyone will first think of Dubbo’s service registration center ZooKeeper. However, many readers often have a question: Why doesn’t RocketMQ directly use ZooKeeper as the registration center and choose to implement its own Nameserver registration center?

Let’s first talk about Dubbo’s service registration center, ZooKeeper. A prominent feature of the registration center based on ZooKeeper is dynamic service changes, which can be instantly perceived by consumers. For example, in Dubbo, when a service is dynamically expanded by adding a batch of message service providers, consumers can immediately perceive the change and load new requests to the new service providers. This mode is called the PUSH mode in the industry.

2

The registration center based on ZooKeeper mainly benefits from ZooKeeper’s event mechanism, with the following main processes:

  1. When a message service provider starts, it registers with the registration center. This mainly involves creating an ephemeral node under the /dubbo/{serviceName}/providers directory. When a service provider crashes, the node will be deleted because the session is closed.
  2. When a message consumer starts, it subscribes to a certain service, which actually means creating an ephemeral node under /dubbo/{serviceName}/consumers and monitoring /dubbo/{serviceName}/providers at the same time. If nodes are added or deleted under this node, the consumer will receive an event. ZooKeeper will push the information of all current child nodes under providers to the consumer, and the consumer will update its local cache with the latest service provider list in real time.

The biggest advantage of the registration center based on ZooKeeper is its real-time performance. However, its internal implementation is very complex. ZooKeeper is based on the CP model, which means strong consistency. For strong consistency, availability usually needs to be compromised. For example, if the ZooKeeper cluster triggers re-election or there is a network partition, the entire ZooKeeper cluster will be unable to provide new registration and subscription services, affecting user experiences.

In the field of service registration, data consistency is not so important. For example, in the scenario of service registration and subscription in Dubbo, even if the client (message consumer) obtains inconsistent service provider lists, it will not cause serious consequences. At most, there will be an unbalanced load on the service providers for a period of time. As long as eventual consistency is achieved, it is sufficient.

PULL Mode #

RocketMQ’s Nameserver does not use a registration center like ZooKeeper. Instead, it chooses to implement its own Nameserver registration center. If you have ever looked at the source code of RocketMQ, you will find that this module consists of only 5 to 6 classes with a total code size of no more than 5000 lines. Simplicity means efficiency. The following is an example diagram of the registration center based on PULL mode. 3

  1. The Broker sends a heartbeat packet to the Nameserver every 30 seconds, which includes the routing information of the topic (such as the number of read and write queues and the operation permissions). The Nameserver updates the routing information of the topic through a HashMap and records the timestamp of the last received heartbeat packet from the Broker.
  2. The Nameserver clears the crashed Brokers every 10 seconds. The Nameserver determines that a Broker has crashed if the current system timestamp minus the timestamp of the last received heartbeat packet from the Broker is greater than 120 seconds.
  3. The message producer pulls the routing information of the topic every 30 seconds. This means that the message producer does not immediately perceive the addition or removal of Broker servers.

One typical feature of the PULL pattern is that even if the routing information stored in the registry changes, the client cannot perceive it in real-time and can only rely on periodic tasks to update it. This can cause some problems. For example, after a large promotion, the cluster needs to be scaled down and taken offline. If the process is stopped directly and the network connection is severed, the Nameserver can immediately perceive the offline status of the Broker and store the routing information in memory. However, it will not be immediately pushed to the Producer or Consumer, but will wait until the Producer updates the routing information to the Nameserver periodically. Before the update, when performing message queue load balancing, the queues on the already offline Broker will be selected, resulting in message sending failures.

In RocketMQ, the Nodes in the Nameserver cluster do not communicate with each other and are independent. This implementation is very simple, but it also brings up a problem: the routing information of the Topics will be inconsistent on different Nodes.

So how does Nameserver solve these two problems? The solution adopted by the designers of RocketMQ is not to solve them. In order to ensure the high performance of the Nameserver, these defects are allowed to exist, and it is up to the users to solve them.

Due to the inability of the message sender to perceive the changes in routing information in a timely manner, a message sending retry and fault tolerance mechanism is introduced to ensure the high availability of message sending. This part has been explained in detail in previous articles.

Will the inconsistency of data between Nameservers cause significant problems?

Analysis of the Impact of Nameserver Data Inconsistency #

In RocketMQ, message senders and consumers in the same period of time only connect to one machine in the Nameserver cluster. For example, message sender A may connect to Nameserver-1, while consumer C1 may connect to Nameserver-1 and consumer C2 may connect to Nameserver-2. Let’s talk about the impact of data inconsistency on message sending and consumption.

Example of Nameserver data inconsistency:

4

Impact on the message sender #

As shown in the diagram above, Producer-1 connects to Nameserver-1, while Producer-2 connects to Nameserver-2. For example, both of these message senders need to send messages to the “order_topic” topic. Due to the inconsistent routing information stored in the Nameserver, the impact on message sending is not significant, but it will cause an imbalance in message distribution. Most messages will be sent to broker-a. As long as there is no network partition, the data in the Nameserver will eventually become consistent, and the problem of data imbalance will be resolved quickly. Therefore, from the perspective of the message sender, the inconsistency of routing data in the Nameserver does not cause serious problems.

Impact on the message consumer #

If a consumer group “order_consumer” has two consumers C1 and C2, the routing information obtained by C1 and C2 will be inconsistent due to their connection to different Nameservers. What problems will this cause? In RocketMQ’s PUSH mode, automatic load balancing of message queues is performed. Let’s take the average distribution algorithm as an example to see the load balancing of queues.

  • C1: When querying the number of queues for the “order_topic” during message queue load balancing, it finds that there are 8 queues (2 on broker-a and 2 on broker-b) and 2 online consumers in the consumer group. According to the average distribution algorithm, 4 queues are assigned to C1, which are broker-a: q0, q1, q2, q3.
  • C2: When querying the number of queues for the “order_topic” during message queue load balancing, it finds that there are 4 queues (broker-a), and 2 online consumers in the consumer group. According to the average distribution algorithm, 2 queues are assigned to C2. Since C2 is in the second position in the consumer list, it is assigned to broker-a: q2, q3.

The problem is clear: queues on broker-b cannot be assigned to consumers, and the two consumers will consume the q2 and q3 queues on broker-a at the same time, causing duplicate message processing. If the consumer implements idempotent processing, it will not cause significant impact. However, some queues may not be processed, but with monitoring mechanisms, this situation can be quickly detected and human intervention can be notified.

Of course, as the routing information of the Nameserver becomes consistent, the routing information maintained by all consumers in the same consumer group will be consistent. This means that the message consumption queues will eventually be evenly distributed. As long as the consumer implements idempotent processing, the impact is controllable and will not cause immeasurable losses. It is because of this reason that the designers of RocketMQ, in order to achieve simplicity and efficiency, allow for some defects in the design of the Nameserver. This serves as a good demonstration when making architectural design decisions, not requiring everything to be perfect but understanding how to make choices and trade-offs.