33 About Networking in Linux You Must Know These Part One

33 About Networking in Linux You Must Know These Part One #

Hello, I’m Ni Pengfei.

In the previous sections, we have learned about the working principles of the file system and disk I/O, as well as the corresponding performance analysis and optimization methods. Next, we will move on to the next important module - the network subsystem of Linux.

Due to the complexity of the network processing flow, which is closely related to the process scheduling, interrupt handling, memory management, and I/O that we have discussed earlier, I have chosen to explain the network module as the last resource module.

Similar to the CPU, memory, and I/O, the network is also the core functionality of the Linux system. Networking is a technology that connects different computers or network devices. It is essentially a form of inter-process communication, especially for inter-system communication, which must be done through the network. With the popularity of high concurrency, distributed computing, cloud computing, microservices, and other technologies, network performance has become increasingly important.

So, how does Linux networking work? What are the performance metrics for measuring network performance? In the next two articles, I will guide you through learning about the working principles of Linux networking and the performance metrics.

Network Models #

When it comes to networks, I think you often mention seven-layer load balancing, four-layer load balancing, or devices at the third and second layers, and so on. So what exactly do the second, third, fourth, and seventh layers mean?

In fact, these layers all come from the Open System Interconnection Reference Model, abbreviated as the OSI network model, established by the International Organization for Standardization (ISO).

In order to solve the compatibility issues of heterogeneous devices in network interconnection and decouple the complex network packet processing flow, the OSI model divides the framework of network interconnection into seven layers: the application, presentation, session, transport, network, data link, and physical layers. Each layer is responsible for different functions. Among them,

  • The application layer is responsible for providing a unified interface for applications.

  • The presentation layer is responsible for converting data into formats compatible with the receiving system.

  • The session layer is responsible for maintaining communication connections between computers.

  • The transport layer is responsible for adding a transmission header to the data and forming a data packet.

  • The network layer is responsible for routing and forwarding data.

  • The data link layer is responsible for MAC addressing, error detection, and error correction.

  • The physical layer is responsible for transmitting data frames in a physical network.

However, the OSI model is still too complex and has not provided an achievable method. Therefore, in Linux, we actually use another more practical four-layer model called the TCP/IP network model.

The TCP/IP model divides the framework of network interconnection into four layers: the application, transport, network, and network interface layers. Among them,

  • The application layer is responsible for providing a set of applications to users, such as HTTP, FTP, DNS, etc.

  • The transport layer is responsible for end-to-end communication, such as TCP, UDP, etc.

  • The network layer is responsible for encapsulation, addressing, and routing of network packets, such as IP, ICMP, etc.

  • The network interface layer is responsible for the transmission of network packets in the physical network, such as MAC addressing, error detection, and transmission of network frames through network cards.

To help you have a more vivid understanding of the relationship between TCP/IP and the OSI model, I have drawn a diagram as shown below:

Of course, although Linux implements the network protocol stack based on the TCP/IP model, in our usual learning and communication, we are accustomed to using the OSI seven-layer model to describe it. For example, when it comes to seven-layer and four-layer load balancing, they correspond to the application layer and transport layer in the OSI model (while in the TCP/IP model, they actually correspond to the fourth and third layers).

The TCP/IP model includes a large number of network protocols, and the principles of these protocols are also essential knowledge that everyone must master. If you are not familiar with them, I recommend that you study Volume 1 and Volume 2 of “TCP/IP Illustrated” or learn from the column “Fun with Network Protocols” produced by Geek Time.

Linux Network Stack #

With the TCP/IP model, data packets are processed layer by layer according to the protocol stack when performing network transmission; then, the packet is encapsulated with the protocol header of the corresponding layer and sent to the next layer.

Of course, the processing logic at each layer of the network packet depends on the network protocols used at each layer. For example, in the application layer, an application that provides a REST API can use the HTTP protocol to encapsulate its JSON data into the HTTP protocol and pass it down to the TCP layer.

The encapsulation is simple, just adding fixed-format metadata before and after the original payload, without modifying the original payload data.

For example, taking a network packet communicated via the TCP protocol as an example, we can see from the following figure the encapsulation format of application-layer data at each layer.

In this figure:

  • The transport layer adds a TCP header before the application layer data;

  • The network layer adds an IP header before the TCP packet;

  • And the network interface layer adds a frame header and a frame trailer before and after the IP packet respectively.

These added headers and trailers are filled in accordance with specific protocol formats. To learn about the specific format, you can refer to the documentation of the protocol. For example, you can refer to here to learn about the format of the TCP header.

These added headers and trailers increase the size of the network packet, but as we all know, not any size of packet can be transmitted in a physical link. The maximum transmission unit (MTU) set by the network interface configuration determines the maximum size of IP packets. In the most commonly used Ethernet, the default value of MTU is 1500 (which is also the default value of Linux).

Once the network packet exceeds the size of MTU, it will be fragmented at the network layer to ensure that the resulting IP packets are not larger than the MTU value. Obviously, the larger the MTU, the fewer the number of fragments required, and the better the network throughput.

After understanding the TCP/IP network model and the encapsulation principle of network packets, you can easily imagine that the network stack in the Linux kernel is actually similar to the four-layer structure of TCP/IP. The following figure is an illustration of the Linux General IP network stack:

(Reference to the picture “10.7 General IP Network Stack” in the book “The Art of Performance Tuning”)

If we look at this network stack from top to bottom, we can see that:

  • The uppermost layer, the application program, needs to interact with the socket interface through system calls;

  • Below the socket is the transport layer, the network layer, and the network interface layer we mentioned earlier;

  • The bottommost layer consists of the network card driver program and the physical network card device.

Here I’ll briefly talk about the network card. The network card is the basic device for sending and receiving network packets. During system startup, the network card is registered with the system through the network card driver program in the kernel. During the process of network reception and transmission, the kernel interacts with the network card through interrupts.

Combined with the Linux network stack mentioned earlier, we can see that the processing of network packets is very complex. Therefore, the network card hardware interrupts only handle the most important network card data reading or sending, and most of the logic in the protocol stack is handled in software interrupts.

Linux Network Receive and Transmit Process #

After understanding the Linux network stack, let’s take a look at how Linux receives and transmits network packets.

Note that the following content is based on physical network cards. In fact, Linux also supports numerous virtual network devices, and their network receive and transmit processes may have some differences.

Network Packet Receive Process #

Let’s start by looking at the network packet receive process.

When a network frame arrives at the network card, the card places the network packet into the receive packet queue using DMA. Then, through a hardware interrupt, the network card tells the interrupt handler that it has received a network packet.

Next, the network card interrupt handler allocates a kernel data structure (sk_buff) for the network frame and copies it to the sk_buff buffer. Then, through a software interrupt, it notifies the kernel that a new network frame has been received.

The kernel protocol stack then retrieves the network frame from the buffer and processes it layer by layer from bottom to top through the network protocol stack. For example,

  • At the link layer, it checks the validity of the message, identifies the upper-layer protocol (such as IPv4 or IPv6), removes the frame header and trailer, and then hands it over to the network layer.

  • At the network layer, it extracts the IP header, determines the next step for the network packet (such as whether it should be handled by the upper layer or forwarded). When the network layer confirms that the packet is destined for the local host, it extracts the upper-layer protocol type (such as TCP or UDP), removes the IP header, and then hands it over to the transport layer.

  • At the transport layer, after extracting the TCP or UDP headers, it uses the <source IP, source port, destination IP, destination port> four-tuple as an identifier to find the corresponding socket. It then copies the data to the receive buffer of the socket.

Finally, the application program can use the Socket interface to read the newly received data.

To illustrate this process more clearly, I have created a diagram. The left half of the diagram represents the receive process, and the pink arrows in the diagram represent the processing path of the network packet.

Network Packet Transmit Process #

After understanding the network packet receive process, it is easy to understand the network packet transmit process. The network packet transmit process is the right half of the diagram. It is easy to see that the direction of the packet transmission is opposite to the receive direction.

First, the application program calls the Socket API (such as sendmsg) to send the network packet.

Since this is a system call, it will transition to the kernel mode socket layer. The socket layer places the packet in the Socket send buffer.

Next, the network protocol stack retrieves the packet from the Socket send buffer and processes it layer by layer from top to bottom according to the TCP/IP stack. For example, the transport layer and the network layer add the TCP header and IP header, respectively. They perform routing lookups to determine the next-hop IP and perform fragmentation based on the MTU size.

The fragmented network packet is then sent to the network interface layer for physical address resolution to find the MAC address of the next-hop. Then, the layer adds the frame header and trailer and places it into the transmit queue. Once this is done, a software interrupt notifies the driver that there are new network frames in the transmit queue that need to be sent.

Finally, the driver reads the network frame from the transmit queue using DMA and sends it out through the physical network card.

Summary #

In today’s article, I have explained the working principles of Linux networking.

Multiple servers are interconnected through network devices such as network cards, switches, and routers, forming a network. Due to the heterogeneity of network devices and the complexity of network protocols, the International Organization for Standardization has defined a seven-layer OSI network model. However, this model is too complex and in practical work, the more practical TCP/IP model is used as the standard.

The TCP/IP model divides the framework of network interconnection into four layers: the application layer, transport layer, network layer, and network interface layer. These are also the core components of the Linux network stack.

  • When an application sends a data packet through the socket interface, it goes through the network protocol stack from top to bottom for layer by layer processing, and is eventually sent out through the network card.

  • When receiving data, it goes through the network stack from bottom to top for layer by layer processing, and is eventually delivered to the application.

Now that you understand the basic principles and the process of sending and receiving data in Linux networking, you must be eager to know how to observe the performance of the network. So, specifically, what metrics can measure the performance of Linux networking? Don’t worry, I will explain this to you in detail in the next section.

Reflection #

Finally, I would like to discuss your understanding of Linux networking. What network-related performance bottlenecks have you encountered? How do you analyze them? You can combine the network knowledge you have learned today to present your own views.

Feel free to discuss with me in the comments section, and also feel free to share this article with your colleagues and friends. Let’s practice in real-world scenarios and improve through communication.