01 Message Broker System ABC #

01 Message Engine System ABC

Hello, I am Hu Xi. Welcome to the “Kafka Core Technology and Practice” column. If you are interested in Kafka and the underlying message engine and stream processing, I am happy to have you here and learn about all aspects of Kafka together in the coming days.

Undoubtedly, you must now be full of curiosity about Apache Kafka, so allow me to try to answer the question of what Kafka is. By the way, I will continue to answer this question in the next edition, and the answer will be different. So, what is Kafka? In a nutshell: Apache Kafka is an open-source message engine system.

If the term “message engine system” is unfamiliar to you, then I’m sure you have heard of the terms “message queue” and “message middleware”. But to be honest, I prefer to use the term “message engine system” because “message queue” implies something unclear, as if Kafka is built using queues, and the term “message middleware” is exaggerated and makes it unclear what exactly this middleware is used for.

Systems like Kafka are called “Messaging Systems” in foreign countries, and many Chinese literature translations simply refer to them as “message systems”. Personally, I think it is not very appropriate because it emphasizes the role of the message body unilaterally, while neglecting the messaging attributes that these systems are proud of. Just like an engine, it has the ability to convert and transmit certain types of energy, so I think the translation “message engine” is more appropriate.

Speaking of this, let’s digress a little. I think that currently, the standardization of translating foreign proprietary technical terms is not enough in China, and there are all kinds of names and terms. Let me give you an example, such as the famous Raft algorithm and Paxos algorithm. Those who are familiar with them know that their purpose is to achieve consensus among multiple nodes on a certain decision in a distributed system, and they both belong to the Consensus Algorithm family. If you search for the Raft algorithm in a search engine, in China it is mostly referred to as a “consistency algorithm”. In fact, I think translating it as a “consensus algorithm” is the most accurate. We use the word “consistency” too frequently, and the English term “Consistency” is translated as “consistency”, “Consensus” is also translated as “consistency”, and even “Coherence” is translated as “consistency”.

Let’s get back on track and continue talking about message engine systems. What are these systems used for? Let me start with an official and serious version of the answer.

According to the definition from Wikipedia, a message engine system is a set of specifications. Enterprises use this set of specifications to pass semantically accurate messages between different systems, achieving loosely coupled asynchronous data transmission.

Indeed, this is an official definition, precise and to the point. If you find it difficult to understand, then you can try my folk version below:

System A sends messages to the message engine system, and system B reads the messages sent by A from the message engine system.

The most basic message engine system does this! Regardless of which version above, they both mention two important facts:

• The object being transmitted by the message engine is a message;
• How to transmit messages is part of the message engine’s design mechanism.

Since the message engine is used to transmit messages between different systems, how to design the format of the messages to be transmitted has always been a top priority. How can a message achieve unambiguous expression of business semantics and at the same time provide maximum reusability and generality? Take a few seconds to pause and think about it. If it were you, how would you design your message encoding format?

One relatively easy-to-think-of approach is to use existing mature solutions, such as using CSV, XML, or JSON; or you may be familiar with some serialization frameworks open-sourced by foreign giants, such as Google’s Protocol Buffer or Facebook’s Thrift. These are all cool options. Now let me tell you Kafka’s choice: it uses a pure binary byte sequence. Of course, the messages are still structured, but they must be converted into binary byte sequences before use. After the message design is completed, it is not enough. The message engine system also needs to set the specific transmission protocol, that is, how to send the message. There are two common methods:

• Point-to-Point Model: Also known as the message queue model. If we use the aforementioned “folk version” definition, messages sent by System A can only be received by System B, and no other system can read the messages sent by A. A daily-life example of this model is telephone customer service: only one customer service representative can handle incoming calls from the same customer; the second representative cannot serve the same customer.
• Publish/Subscribe Model: The difference from the above model is that it introduces the concept of a topic, which can be understood as a message container with logical semantics. This model also has a sender and a receiver, but the terminology is different. The sender is also called a publisher, and the receiver is called a subscriber. Unlike the point-to-point model, this model may have multiple publishers sending messages to the same topic, and there may also be multiple subscribers who can receive messages from the same topic. Newspaper subscriptions in real life are typical examples of the publish/subscribe model.

What’s cool is that Kafka supports both of these message engine models. I will share how Kafka achieves this later in this column.

You may wonder about the relationship between JMS and message engine systems. JMS stands for Java Message Service, which also supports these two message engine models mentioned above. Strictly speaking, JMS is not a transmission protocol, but simply a set of APIs. However, JMS is so well-known that many mainstream message engine systems support the JMS specification, such as ActiveMQ, RabbitMQ, IBM WebSphere MQ, and Apache Kafka. Of course, Kafka does not fully conform to the JMS specification. On the contrary, it takes a unique path and explores its own way.

Okay, so far we have only understood what message engine systems do and how they work, but there is still an important question: why should we use them?

Taking the aforementioned “folk version” as an example, we can’t help but ask, why can’t System A directly send messages to System B without the intermediate message engine?

The answer is “peak shaving and valley filling”. These four words are even more famous than the message engine itself. I have read a lot of literature, and the most common phrase is these four characters. The so-called “削峰填谷” refers to buffering the instantaneous surge flow in upstream and downstream to make it smoother. Especially for upstream systems with strong sending capabilities, if there is no protection from a message engine, the “fragile” downstream system may be directly overwhelmed, resulting in a “cascade” failure of the entire chain. However, once there is a message engine, it can effectively counter the impact of upstream flow, truly filling the upstream “peak” into the “valley”, avoiding flow fluctuations. Another advantage of message engine systems is the loose coupling between senders and receivers, which also simplifies application development to a certain extent and reduces unnecessary interactions between systems.

After saying so much, you may not have a very intuitive understanding of “削峰填谷”. Let me give you an example of how Kafka “resists” peak traffic in this regard. Think back to how you purchased this course on Geek Time. If I remember correctly, each course on Geek Time has a dedicated subscribe button, which, when clicked, takes you to the payment page. This simple process may involve multiple sub-services, such as clicking the subscribe button triggers the order system to generate the corresponding order, and processing the order involves calling multiple downstream sub-system services, such as calling the Alipay and WeChat Pay interfaces, querying your login information, and verifying course information, etc. Obviously, the upstream order operation is relatively simple, and its TPS is much higher than that of the downstream order processing service. Therefore, if the upstream and downstream systems are directly connected, the downstream service may fail to process the upstream orders in a timely manner, resulting in a backlog of orders. Especially when there are business activities like flash sales, the upstream order flow will increase instantly, which may directly overwhelm the downstream sub-system services.

One common solution to this problem is to limit the speed of the upstream system. However, this approach is clearly unreasonable for the upstream system, as the problem does not lie there. So the more common approach is to introduce message engine systems like Kafka to counteract this TPS mismatch and instantaneous peak traffic between upstream and downstream systems.

In this example, when Kafka is introduced, the upstream order service no longer interacts directly with the downstream sub-services. After a new order is generated, it simply sends an order message to Kafka Broker. Similarly, each downstream sub-service subscribes to the corresponding topic in Kafka and obtains order messages from their respective partitions in real-time for processing. This effectively decouples the upstream order service from the downstream order processing service. Therefore, when flash sales occur, Kafka can save the transient increase in order traffic in the corresponding topics as messages, without affecting the TPS of the upstream service, while leaving enough time for the downstream sub-services to consume them. This is the greatest significance of message engine systems like Kafka.

If you are not familiar with terms such as Kafka Broker, topics, and partitions, there is no need to worry. I will take the time to introduce the common concepts and terms of Kafka later in this column.

Before we end today, I want to share a little story with you. Back in 2015, it took me nearly a year to read the Kafka source code, and I often wanted to give up during that time. You know how painful it is to read nearly 500,000 lines of code. I remember that I filled a thick notebook with handwritten source code comments. Luckily, I persevered, and all the previous efforts were not in vain. That’s why writing books and columns for Geek Time became a natural thing for me later on.

Lastly, I want to give you a sentence: “Even smart people have to work hard.” I don’t remember if this was said by Zeng Guofan or Ji Xianlin, but this sentence has had a great impact on me. When I feel restless, it helps me calm down and focus on doing things. I hope it inspires you as well. Remember: smart people have to work hard!

Open Discussion #

Please discuss your understanding of messaging engine systems or share how your company or organization uses messaging engines to address practical problems.

Feel free to write down your thoughts and answers, and let’s discuss together. If you find it insightful, you are also welcome to share this article with your friends.