00 Preface How to Make the Linux Kernel Serve Applications Better

00 Preface How to Make the Linux Kernel Serve Applications Better #

Hello, I’m Shaoyafang. Welcome to my course, where we will learn about the Linux kernel together.

Since 2010, I have been working with the Linux kernel and have been involved in solving a large number of performance issues directly related to production environments. A few years ago, while working at Mogujie, the business was growing rapidly. As you know, with business growth comes challenges in maintaining service stability. For example, how to analyze TCP retransmission, how to troubleshoot memory leaks without interrupting tasks at runtime, and how to quickly resolve high CPU sys utilization. These are real problems that you will either know how to handle or not.

Let’s take TCP retransmission as an example. If you are familiar with it, you would usually monitor the TCP retransmission rate on servers, as shown in the following figure:

As shown in the picture, such a high TCP retransmission rate will undoubtedly result in a decrease in system QPS. So you cannot be lax; you need to quickly identify where the problem lies. But when you really investigate, you will find that you don’t know where to start because the on-site information when retransmission occurs is not recorded. The reason these on-site information is not recorded is that the cost of recording is too expensive. If you have troubleshooted network issues before, you should know that network data can be very large. Even if only the TCP header information is recorded, it still incurs significant storage overhead, and the process of recording information also brings a lot of performance overhead.

To address this type of problem, our team has made many improvements to the monitoring of TCP retransmission, which can improve the team’s efficiency in locating and analyzing problems without significantly affecting business performance. The TCP retransmission on-site information we record is as follows:

18:21:58 10.17.130.19:20612 124.74.250.144:44  SYN_SENT   
18:22:00 10.17.130.19:20612 124.74.250.144:443 SYN_SENT   
18:23:21 10.17.130.19:20716 124.74.250.144:443 SYN_SENT   
18:23:23 10.17.130.19:20716 124.74.250.144:443 SYN_SENT   
18:24:39 10.17.130.19:20796 124.74.250.144:443 SYN_SENT   
18:24:41 10.17.130.19:20796 124.74.250.144:443 SYN_SENT   
18:25:43 10.17.130.19:20861 124.74.250.144:443 SYN_SENT   
18:25:45 10.17.130.19:20861 124.74.250.144:443 SYN_SENT   

18:27:23 10.17.130.19:20973 124.74.250.144:443 SYN_SENT   
18:27:25 10.17.130.19:20973 124.74.250.144:443 SYN_SENT

From the above information, you can easily see where TCP retransmission occurs between which servers (IP addresses), which services (server ports), and why it is being retransmitted (SYN_SENT). You see, this increases efficiency, right? I will explain in detail how we actually do it in our course.

In my opinion, for locating complex stability issues like TCP retransmission, it is not enough to analyze from the perspective of developers. It is important to analyze from the system and kernel perspectives to truly trace the root cause and solve the problem once and for all.

The difficulty people face with these problems is ultimately due to a lack of understanding of the Linux kernel. For example, most business developers I have come across have been troubled by performance issues related to their business. However, when analyzing these performance issues, many can only analyze the spikes caused by system calls, while some experts can go deeper to identify the specific system resources causing the spikes. Another example is when TCP retransmission occurs, some can see from the information in tcpdump which TCP connection caused the retransmission, but experts can understand why the retransmission happened based on this information.

Those who can deeply analyze problems at the Linux kernel level have strong problem-solving skills and a keen ability to pinpoint and analyze issues. As a result, they can solve problems that others cannot, and are often the top performers in their respective fields.

However, most application developers often neglect learning about the Linux kernel, which is understandable. After all, their main focus is on optimizing and deploying business code. In the environment of long working hours (commonly known as “996”) in internet companies, it is hard to find the time and energy to delve into the kernel. Furthermore, the Linux kernel itself is very complex, and this complexity can deter application developers as well as beginners in kernel development.

Speaking for myself, in order to study the Linux kernel, I have read many technical books in both Chinese and English, such as:

  • To understand how applications use the Linux kernel, I have read “Advanced Programming in the UNIX Environment” multiple times.
  • To master system architecture, I carefully read “Computer Systems: A Programmer’s Perspective” and “The Architecture of Computer Hardware and Systems Software”.
  • To understand how applications are compiled on Linux, I deeply studied “An Introduction to GCC” and “Linkers and Loaders”.
  • To master the principles of Linux device drivers, I read “Linux Device Drivers” and then started reading “Understanding the Linux Kernel”, which is an introductory book for Linux kernel developers.
  • In the learning process, to better understand the memory subsystem, I read “Understanding the Linux Virtual Memory Manager”.
  • To master the network subsystem, I read “TCP/IP Illustrated, Volume 1: The Protocols” and “TCP/IP Architecture, Design and Implementation in Linux”.
  • To familiarize myself with the design principles of other operating systems, I read “The Design and Implementation of the FreeBSD Operating System” to learn about FreeBSD.
  • To better analyze Linux kernel issues, I read “Debug Hacks: Tips & Tools for Finding and Fixing Bugs” and “Systems Performance: Enterprise and the Cloud”.
  • In order to better communicate with overseas developers, I have read a large number of original English books to improve my English proficiency. The book titles I listed above are some of the original technical books in English.

There is no denying that the cost of systematically learning the Linux kernel is very high. However, if you are not a kernel developer, there is really no need to understand every detail, master every mechanism, and comprehend all the excellent design ideas of the kernel.

In my opinion, the essence of an excellent software or code is to solve the real problems we encounter and better meet our actual needs.

That is to say, it is enough to solve practical application layer problems through the Linux kernel knowledge you have mastered, which is also the original intention of offering this course. I hope to pass on my many years of learning and practical experience in the Linux kernel to you by “solving problems and meeting needs”, so that the Linux kernel can better serve your applications.

To achieve this goal, I will start with some common problems in production environments and help you understand: how your application interacts with system resources? What kind of configuration is better for your business type? How to troubleshoot tricky problems step by step?

So, from the perspective of system resources, the problems we need to pay attention to can be divided into four categories: disk I/O, memory, network I/O, and CPU. In this series of courses, I will take you deep into the typical problems in these four categories: “Page Cache management problem, memory leakage problem, TCP retransmission problem, kernel CPU utilization skyrocketing problem”. These also correspond to the four modules of our course.

By mastering these four typical problems and their analysis methods, you will have a deeper understanding of disk I/O, memory, network I/O, and CPU, the four basic resources on servers. You will also be able to apply this knowledge to solve other problems and no longer shy away from challenging system problems.

To facilitate your progressive learning, each module of our course will be presented in three parts: basics, case studies, and analysis.

In the “Page Cache management” module, I will focus on how to better utilize Page Cache to reduce unnecessary I/O overhead, the problems caused by improper Page Cache management, and how to analyze and solve these problems.

In the “memory leakage” module, I will focus on analyzing how application programs allocate and release memory from the system. Through memory leak cases, I will help you understand the details of memory usage in application programs and the problems that can be caused by improper memory usage. Of course, I will also guide you to observe, analyze, and solve these problems.

In the “TCP retransmission” module, I will focus on analyzing the process of TCP connection establishment, transmission, and termination. What configuration items affect this process? What network problems can be caused by improper configuration? Then, starting from specific cases of TCP retransmission, I will help you understand some network details that you must master, as well as how to analyze and solve network-related issues when you encounter them. In the High CPU Usage in Kernel State module, I will analyze how applications can efficiently use the CPU and the scenarios that can lead to inefficient CPU usage. For example, high CPU usage in the kernel state is a clear sign of low efficiency. In this case, I will focus on explaining the Linux kernel features or system configuration options that can cause this problem, as well as how to analyze and solve specific issues.

At the end of each module, I will summarize the general analytical approach to common problems, so that you can have a rough direction when facing similar issues.

Of course, this course is not only for application developers and operators but also helpful for kernel developers, especially those who are not very experienced. It can help you better understand the business aspect. The Linux kernel is essentially designed to serve businesses, and understanding the business aspect is essential in implementing kernel features effectively. When I submit patches to the Linux kernel, maintainers always like to ask, “What is your real-life use case?” Combining the business aspect is something every Linux kernel developer should keep in mind – don’t design blindly or overly, and be guided by the understanding of the business.

This course not only teaches you how to learn the Linux kernel better but also encourages you to critically analyze the Linux kernel.

Since Linus first released the Linux kernel in 1991, it has evolved for nearly 30 years, and the codebase has grown from the initial 10,000 lines of code to tens of millions of lines of code today. With such complexity and even some bloating, bugs are inevitable, and there are also many poor design choices. So, many puzzling phenomena you encounter in your daily work may actually be Linux kernel defects. In this course, we will also introduce you to these common defects that are everywhere.

When you encounter doubts or think that the Linux kernel is not good enough, feel free to question it, seek help from knowledgeable people in the Linux kernel field, or ask the Linux community for assistance. It is your needs that drive us to build a better and more powerful Linux kernel. Although Linus himself has a bit of a temper, the overall Linux community is very friendly. Regardless of the difficulties you encounter, there will always be someone willing to answer your questions and even guide you on how to improve the Linux kernel.

Finally, let me give you a brief introduction to my work experience in the Linux kernel field. I started working on Linux kernel development at Huawei in 2010. Later, I also worked for a multinational corporation (Juniper Networks), an internet company (Mogujie.com), and my current employer, a well-known internet company.

Different corporations have their own styles, different business scenarios, but they all have the same demand for the Linux kernel: to be better and more stable.

These different business scenarios have enriched my understanding of the Linux kernel. I gradually realized the shortcomings of the Linux kernel in dealing with more and newer business scenarios, so I started getting involved in the Linux kernel community to improve it.

Currently, I mainly work in the memory management subsystem of the Linux kernel ( [email protected] ). If you follow this mailing list, you will often see my name (Yafang Shao < [email protected] >). If you encounter any Linux kernel-related issues in your work, you can also send me an email directly. However, because I work “996” during weekdays, I can only reply to emails on weekends with high probability. Of course, you are also welcome to leave a comment or ask questions under this course, and I will address them as soon as possible.

Finally, I want to remind you again that learning Linux low-level knowledge is not a one-time task, and it takes a lot of time to master it. However, if an experienced person guides you, your learning time will be greatly reduced, and your learning cost will also be significantly reduced. I believe that through this course, you can not only master the essential Linux knowledge but also learn many skills for solving practical problems and avoid stepping into pitfalls that others have already experienced.

Well, welcome to discuss with me. What are the challenges you are currently facing in your work? How well do you understand the Linux kernel? You can record your starting point or questions, and review them after completing the whole course. I believe you will have a different experience.