30 How Tos How to Quickly Analyze System Io Bottlenecks

30 How-tos How to Quickly Analyze System IO Bottlenecks #

Hello, I am Ni Pengfei.

In the previous sections, we analyzed various common I/O performance issues through several cases. Through these practical operations, you should already be familiar with the analysis and locating approach for I/O performance issues, and have mastered many I/O performance analysis tools.

However, I think you may still be confused about how to choose the various performance indicators and tools mentioned in the cases when you are working in different real-world scenarios outside of this column.

In the last section, I left you with an assignment to organize your thoughts. Today, I will review it with you and summarize how to quickly and accurately locate the I/O bottlenecks in a system. We will also clarify how to select indicators and tools in different scenarios, and how to locate performance bottlenecks.

Performance Metrics #

As usual, let’s first review the performance metrics for I/O. You can first recall the principles of file systems and disk I/O, and then write on your own based on your memory and understanding, combined with the I/O stack diagram of the Linux system below. Alternatively, you can refer back to the previous article and summarize one by one.

After learning about I/O performance for so long, when it comes to I/O metrics, you should first think about classification descriptions. We need to distinguish between file systems and disks, and use different metrics to describe their performance.

File System I/O Performance Metrics #

Let’s start with the file system.

First and foremost, the most obvious metric is the usage of storage space, including capacity, usage, and remaining space. We usually refer to these metrics as disk space usage because the data of the file system is ultimately stored on the disk.

However, note that these are only the space usage displayed by the file system, not the actual disk usage, as the metadata of the file system also takes up disk space.

Moreover, if you have configured RAID, the usage you see from the file system may differ from the actual disk usage depending on the RAID level. For example, after configuring RAID10, you can see, at most, only half of the total disk capacity from the file system.

In addition to the storage space for the data itself, there is also a often overlooked metric which is the usage of index nodes, including capacity, usage, and remaining amount. If there are too many small files stored in the file system, you may encounter the problem of index node capacity being full.

Secondly, you should think about the cache usage that has been mentioned multiple times before, including page cache, directory entry cache, index node cache, as well as the specific caches of each file system (such as ext4, XFS, etc.). These caches use faster memory to temporarily store file data or the metadata of the file system, thereby reducing the number of accesses to the slow disk.

In addition to the above points, file I/O is also an important performance metric, including IOPS (including r/s and w/s), response time (latency), and throughput (B/s). When evaluating these metrics, the actual read and write characteristics of the files should also be considered. For example, by combining factors such as file size, number of files, and I/O type, the performance of file I/O can be analyzed comprehensively.

Admittedly, these performance metrics are very important. However, unfortunately, Linux file systems do not provide a direct method to view these metrics. We can only observe and evaluate them indirectly through system calls, dynamic tracing, or benchmark testing.

However, in reality, these metrics are easier to see when we examine disk performance because Linux provides more detailed data for disk performance.

Disk I/O Performance Metrics #

Next, let’s specifically look at the performance metrics that can measure disk I/O performance.

In the article on disk I/O principles, I mentioned four core disk I/O metrics.

  • Utilization, which refers to the percentage of time the disk is busy handling I/O requests. High utilization (e.g., exceeding 60%) usually indicates a performance bottleneck in disk I/O.

  • IOPS (Input/Output Per Second), which refers to the number of I/O requests per second.

  • Throughput, which refers to the size of each I/O request per second.

  • Response time, which refers to the interval from when an I/O request is issued to when a response is received.

When examining these metrics, it is important to analyze them in the context of specific I/O scenarios, such as read and write types (sequential or random), read and write ratios, read and write sizes, and storage types (with or without RAID, RAID level, local storage or network storage), etc.

However, there is a major taboo here, which is directly comparing and analyzing I/O performance metrics in different scenarios. This is a common pitfall that you must avoid.

In addition to these metrics, in the previous articles on cache and buffer principles, I repeatedly mentioned that buffer is also an important metric that needs to be mastered. It often appears in the analysis of memory and disk problems.

These metrics for file systems and disk I/O are very useful and need to be proficiently understood. Therefore, I have summarized them into a diagram to help you classify and remember them. You can save and print it for convenient review at any time, or use it as a “metric selection” checklist for I/O performance analysis.

Performance Tools #

After mastering the performance indicators of the file system and disk I/O, we need to know how to obtain these indicators, or in other words, understand the use of tools.

Do you still remember which tools were used in the previous basics and case studies? Let’s review these contents together.

First, in the section on file system principles, I introduced the tool df for viewing the capacity of the file system. It can be used to view the space capacity of the file system data as well as the capacity of the index nodes. As for file system caching, we observe the page cache, directory entry cache, index node cache, and specific file system cache through various sources such as /proc/meminfo, /proc/slabinfo, and slabtop.

Second, in the section on disk I/O principles, we used iostat and pidstat to observe the I/O situation of the disk and processes, respectively. They are the most commonly used tools for I/O performance analysis. With iostat, we can obtain performance indicators such as I/O utilization, throughput, response time, and IOPS of the disk. With pidstat, we can observe the I/O throughput of processes and the latency of block device I/O.

Third, in the case of heavy logging, we first used top to view the CPU usage of the system and found that iowait was relatively high. Then, we used iostat to find the bottleneck of disk I/O utilization and pidstat to identify processes with a large amount of I/O. Finally, through strace and lsof, we discovered the files that the problem process was reading and writing to, ultimately determining the source of the performance problem—it turned out to be excessive logging by the process.

Fourth, in the case of the word popularity of disk I/O latency, we also used top and iostat at first to find that there was an I/O bottleneck on the disk, and pidstat to identify processes with a large amount of I/O. However, when we tried to use the strace command afterward, we surprisingly did not see any write system calls. So, we changed our approach and used the new tools filetop and opensnoop to trace system calls from the kernel and finally found the bottleneck source.

Finally, in the cases of MySQL and Redis, using the same approach, we first used top, iostat, and pidstat to identify and find the bottleneck sources of I/O performance problems, which were mysqld and redis-server. Then, we used strace+lsof to find the files they were reading and writing to.

Regarding the MySQL case, based on the file paths being read and written by mysqld, combined with the principles of MySQL database engine, we not only identified the names of the database and tables, but also discovered a slow query issue. We ultimately solved the performance bottleneck by optimizing the index.

As for the Redis case, based on the files being read and written by redis-server, as well as the TCP Sockets used for network communication, combined with the working principles of Redis, we found that there were problems with the Redis persistence options. From the data of the TCP Socket communication, we also identified unreasonable behaviors by the clients. So, we modified the Redis configuration options and optimized the way clients use Redis to reduce the frequency of network communication and solve the performance problem.

After reviewing so much information at once, do you feel overwhelmed and once again want to lament the complexity of performance tools? In fact, as long as you understand the corresponding system principles, using the tools is not difficult at all.

Connection between Performance Metrics and Tools #

Similar to the previous sections on CPU and memory, I suggest organizing and memorizing information from two different dimensions: metrics and tools.

  • Starting from I/O metrics, it is easier for you to relate performance tools to the working principles of a system and have a macro understanding of performance issues.

  • Starting from performance tools allows you to quickly learn how to use them and efficiently identify the performance metrics you want to observe. Especially when tools are limited, it is important to fully utilize each tool at hand and extract as much information as possible.

The first dimension starts with performance metrics related to file systems and disk I/O. In other words, when you want to view a specific performance metric, you need to know which tools can provide it.

Classify and understand the performance tools based on different metrics. This way, when troubleshooting performance issues, you can clearly identify which tools can provide the desired metrics, rather than blindly trying them out.

Although you don’t need to memorize all the relevant tools, if you can remember the characteristics of each tool corresponding to a specific metric, you will be more efficient and flexible when performing actual operations.

Here, I have created a table that lists the tools that provide I/O performance metrics. It can help you organize the relationships and facilitate your understanding and memorization. You can save and print it out for easy reference. Of course, you can also use it as a guide for “metric tools”.

Now, let’s move on to the second dimension.

The second dimension starts with tools. That is, when you have installed a specific tool, you need to know which metrics it can provide.

This is also very important in practical environments, especially in production environments. This is because in many cases, you do not have permission to install new tool packages, so you need to maximize the use of existing system tools, which requires sufficient understanding of them.

In terms of the usage of each tool, they generally support a wide range of configuration options. But don’t worry, you don’t need to memorize all these configuration options. You just need to know which tools are available and what their basic functions are. When you actually need to use them, you can use the “man” command to consult their manuals.

Similarly, I have compiled a table that lists these commonly used tools to help you differentiate and understand them. Naturally, you can also use it as a guide for “tool metrics” and consult it whenever needed.

How to Quickly Analyze I/O Performance Bottlenecks #

By now, I believe you are already very familiar with memory performance metrics and understand what tools can be used to obtain each performance metric.

You may have noticed that although there are still many I/O performance metrics for the file system and disk in comparison to the previous two sections, the core performance tools are actually just a few. Mastering them proficiently and combining them with the phenomena of the actual system, as well as the principles of the system and application programs, will make I/O performance analysis clear.

However, no matter what, it is not practical to run all the tools mentioned above every time you encounter an I/O performance problem.

In a real production environment, what we hope for is to quickly locate the bottleneck of the system and then optimize the performance as quickly as possible, which means solving performance problems both quickly and accurately.

So, is there a method to quickly and accurately identify the I/O bottleneck of the system? The answer is definitely yes.

The key is to find correlations. There are certain correlations among various performance metrics, so they should not be viewed in isolation. To understand the correlations of performance metrics, it is necessary to have a good understanding of the working principles of each performance metric. This is also why I have interspersed explanations of related system principles when introducing each performance metric. I hope you can remember this point again.

Let’s take the previous cases as examples. If you carefully compare the previous cases, from the case of I/O latency to the cases of MySQL and Redis, you will find that although these problems are quite different, the initial analytical approach is basically the same from the perspective of I/O analysis. They all follow these steps:

  1. Use iostat to identify the disk I/O performance bottleneck;
  2. Then, use pidstat to locate the process causing the bottleneck;
  3. Next, analyze the I/O behavior of the process;
  4. Finally, analyze the source of these I/Os in combination with the principles of the application program.

Therefore, in order to narrow down the scope of investigation, I usually run those tools that support multiple metrics first, such as iostat, vmstat, pidstat, etc. Then, based on the observed phenomena and combined with the principles of the system and application programs, I look for the next step of analysis. I have illustrated this process in a graph, which you can save for reference.

The graph lists several commonly used file system and disk I/O performance analysis tools, as well as the corresponding analysis process. The arrows indicate the direction of analysis. Among these, iostat, vmstat, and pidstat are the most core performance tools, and they also provide the most important I/O performance metrics. Here are a few examples that you may find easier to understand.

For example, in the cases of MySQL and Redis mentioned earlier, we confirmed the disk I/O performance bottleneck using iostat, then used pidstat to find the process with the highest I/O, and then used strace to find the files that the process is reading and writing, and finally combined with the principles of the application program to identify the cause of the high I/O.

Another example is when you find a disk I/O performance bottleneck with iostat, and then use pidstat and vmstat to check, you may find that the I/O comes from kernel threads, such as a significant increase in Swap usage. In this case, you need to perform memory analysis, find the process that consumes a large amount of memory, and then find ways to reduce memory usage.

Also, please note that in this graph, I only listed the most core performance tools and did not include all the tools mentioned in the previous table. I did this to avoid overwhelming you with a long list of tools. It may not be a good thing to be exposed to all core or niche tools at the beginning of learning. On the other hand, I also hope that you can focus on the core tools first, as mastering them will enable you to solve most problems.

Therefore, you can save this graph as a thinking roadmap for file system and disk I/O performance analysis. Start with the most core tools, practice with the cases I provided in actual environments, and master them.

Summary #

Today, we reviewed common file systems and disk I/O performance indicators together, sorted out common I/O performance observation tools, and established the correlation between performance indicators and tools. Finally, we summarized the approach to quickly analyze I/O performance issues.

As the saying goes, although there are many performance indicators for I/O and many corresponding performance analysis tools, once you are familiar with the meanings of each indicator, you will naturally find their correlations. Following this line of thinking and mastering common analysis techniques is not difficult either.

Reflection #

In this column, I have only listed a few of the most common cases to help you understand the principles and analysis methods of file system and disk I/O performance. You must have encountered many other I/O performance issues as well. I would like to discuss with you the I/O performance problems you have encountered and how you analyzed their bottlenecks.

Please feel free to discuss with me in the comments section, and you are also welcome to share this article with your colleagues and friends. Let’s practice in real scenarios and progress through communication.