06 Stick package and unpacking problem, how to obtain a complete network packet #

In this lesson, we will start learning about the encoding and decoding techniques used in Netty communication. Encoding and decoding techniques are the foundation of network communication, allowing us to define application layer protocols that meet our business requirements. In network programming, we often use various network transport protocols, with TCP being the most commonly used protocol. To better understand Netty’s encoding and decoding framework, we first need to understand the basic packet assembly and disassembly problems of TCP and the common solutions.

Why Packet Assembly and Disassembly? #

TCP is a stream-oriented transmission protocol without packet boundaries. When the client sends data to the server, it may break up a complete message into multiple small packets to send, or it may merge multiple messages into one large packet to send. This leads to packet assembly and disassembly.

Why does this packet assembly and disassembly phenomenon occur? During the process of network communication, there are various factors that limit the size of data packets that can be sent at a time, such as the Maximum Transmission Unit (MTU), Maximum Segment Size (MSS), sliding window, etc. If the size of the data packet transmitted in one go exceeds the MTU size, our data may be split into multiple packets for transmission. If each data packet sent is very small and we make a total of 10,000 requests, TCP will not send 10,000 separate packets. This is because TCP optimizes this with the Nagle algorithm. If you are a network novice, you may not be very familiar with these concepts. So, let’s first understand the basic concepts of MTU, MSS, Nagle, and why they cause packet assembly and disassembly problems.

Maximum Transmission Unit (MTU) and Maximum Segment Size (MSS) #

The Maximum Transmission Unit (MTU) is the maximum size of data that can be transferred in a single link layer transmission. The MTU is generally 1500 bytes. Maximum Segment Size (MSS) refers to the maximum length of a TCP segment, which is the maximum amount of data that can be sent in a single transmission at the transport layer. As shown in the figure below, the relationship between MTU and MSS is generally calculated as MSS = MTU - IP header - TCP header. If MSS + TCP header + IP header > MTU, then the packet will be split and sent in multiple packets. This is the phenomenon of packet disassembly.

Sliding Window #

The sliding window is an effective measure used by the TCP transport layer for flow control, also known as the advertisement window. The sliding window is the window size set by the data receiver, and the receiver informs the sender about the window size, thereby limiting the size of data that the sender can send each time, achieving flow control. This allows the data sender to send multiple data segments simultaneously without blocking and waiting for acknowledgment after sending each set of data. The sending data segments are limited to the window size. Consequently, the sliding window can significantly improve network throughput.

So, how does the TCP segment ensure that packets arrive in order and without loss of data? First of all, all data frames are numbered. TCP does not reply with an ACK response for each segment, but replies with ACK once for multiple segments. For example, if there are three segments A, B, and C, and the sender sends B and C first, the receiver must wait for segment A to arrive. If segment A does not arrive within a certain period of time, segments B and C will also be discarded, and the sender will initiate a retry. If segment A arrives, an ACK confirmation will be sent to the sender.

Nagle Algorithm #

The Nagle algorithm was defined as a TCP/IP congestion control mechanism by Ford Aerospace and Communications Corporation in 1984. It is mainly used to solve the network congestion caused by frequent small packet transmission. Imagine if each data packet that needs to be sent is only 1 byte, plus 20 bytes of IP header and 20 bytes of TCP header, the size of each packet is 41 bytes, but only 1 byte is valid information. This results in a significant waste. The Nagle algorithm can be understood as batch sending, which is an optimization approach commonly used in programming. It writes the data into the buffer before it is confirmed, waits for the data to be confirmed or for the buffer to accumulate to a certain size, and then sends the packet.

By default, Linux enables the Nagle algorithm, which can effectively reduce network overhead in scenarios with a large number of small packets. However, if your business scenario requires immediate response for each data sent, the Nagle algorithm cannot meet your requirements because it introduces some data delay. You can disable the Nagle algorithm by using the TCP_NODELAY parameter provided by Linux. In order to minimize the data transmission delay, Netty disables the Nagle algorithm by default, which is opposite to the default behavior of the Linux operating system.

Solution for Packet Splitting or Packet Stickiness #

During the communication between the client and the server, the server may receive an uncertain amount of data at a time. As shown in the above diagram, the following five situations may occur regarding packet splitting or packet stickiness:

• The server happens to read two complete data packets, A and B, without any packet splitting or packet stickiness problems;
• The server receives a data packet that combines A and B together, and it needs to parse out A and B separately;
• The server receives a complete part of data packet B-1 from A and B being stuck together, and it needs to parse out the complete A and wait for the complete B data packet to be read;
• The server receives a partial data packet A-1, and at this point it needs to wait for the complete A data packet to be received;
• Data packet A is large, so the server needs multiple times to receive the complete data packet A.

Due to the existence of packet splitting or packet stickiness problems, the data receiver has difficulty determining where the boundaries of the data packets are, making it hard to identify a complete data packet. Therefore, a mechanism is needed to identify the limits of the data packets, and this is the only solution to solve the problem of packet splitting or packet stickiness: defining the application layer communication protocol. Now let’s take a look at the solutions for mainstream protocols.

Fixed Message Length #

Each data message needs to have a fixed length. When the receiver accumulates a data packet of the fixed length, it considers that it has obtained a complete message. When the data from the sender is smaller than the fixed length, padding is needed.

+----+------+------+---+----+

| AB | CDEF | GHIJ | K | LM |

+----+------+------+---+----+

Assuming our fixed length is 4 bytes, the 5 data records shown above would require 4 packets to be sent:

+------+------+------+------+

| ABCD | EFGH | IJKL | M000 |

+------+------+------+------+

The fixed length method is very simple to use, but it has obvious drawbacks. It cannot set a fixed length value well. If the length is too large, it will waste bytes. If the length is too small, it will affect message transmission. Therefore, the fixed length method is not usually adopted in general cases.

Specific Delimiter #

Since the recipient cannot determine the boundaries of the messages, we can add a specific delimiter to the end of each packet, and the recipient can use this delimiter to split the messages. The following packets, split by the delimiter “\n”, can be parsed into 5 original messages: AB, CDEF, GHIJ, K, and LM.

+-------------------------+

| AB\nCDEF\nGHIJ\nK\nLM\n |

+-------------------------+

Since a specific delimiter needs to be added to the end when sending packets, the choice of delimiter must avoid conflicts with characters in the message body. Otherwise, incorrect message splitting may occur. A recommended approach is to encode the messages, such as using base64 encoding, and then choose a character outside of the 64 encoding characters as the specific delimiter. The specific delimiter method is more efficient in scenarios where the message protocol is simple, such as the famous Redis, which uses a newline delimiter in its communication process.

Message Length + Message Content #

Message Header     Message Body

+--------+----------+

| Length |  Content |

+--------+----------+

Message Length + Message Content is the most commonly used protocol in project development. The example above shows the basic format of this protocol. The message header stores the total length of the message, for example, using a 4-byte int value to record the length, while the message body stores the actual binary byte data of the message. When parsing the data, the recipient first reads the length field “Len” in the message header and then reads byte data with a length of “Len”. This data is considered a complete data message. Taking the original byte data mentioned earlier as an example, the result of encoding using this protocol is as follows:

+-----+-------+-------+----+-----+

| 2AB | 4CDEF | 4GHIJ | 1K | 2LM |

+-----+-------+-------+----+-----+

The Message Length + Message Content protocol is very flexible and does not have the obvious drawbacks of the Fixed Length method and Specific Delimiter method. Of course, the message header can store not only the message length but also other necessary extension fields, such as message version, algorithm type, etc.

Summary #

In this course, we have discussed in detail the issue of packet fragmentation/concatenation in TCP and how to solve this issue using application layer communication protocols. Among them, the variable length protocol based on Message Length + Message Content is the most commonly used method in project development, which we need to focus on. For example, open-source middleware like Dubbo and RocketMQ have customized their own communication protocols based on this method. In the next course, we will learn how to design efficient, scalable, and maintainable custom network communication protocols.