25 FastThread Related Tools Introduction- To Surpass a Thousand Miles, Aim Even Higher #
Introduction to FastThread #
In the previous chapters, we learned that we can print all the thread information of the JVM and then analyze it. However, all the thread information is very long and looks almost the same, which makes it difficult to read each time.
So, whenever I analyze threads, I always think that it would be great if there was a tool that could summarize the general situation and automatically analyze the JVM thread situation for me. FastThread is such a tool.
FastThread is a thread dump analysis tool. The official website is: http://fastthread.io/.
This tool is developed and supported by tier1app. This company now mainly provides three JVM analysis tools, in addition to FastThread, they are:
- GCEasy, visit: https://gceasy.io/, for more details see the article [“Interpretation and Analysis of GC Logs (Extra Edition - Visualization Tools)”].
- HeapHero, visit: https://heaphero.io/, as the name suggests, this is a Heap Dump analysis tool.
FastThread tool can be used for analysis and problem localization. Its features include:
- Universal thread dump analysis. FastThread is a universal thread dump analysis tool that can be used to perform root cause analysis (RCA) by using the thread dumps exported by JVM.
- Provides online analysis functionality. Since thread dumps are generally not too large, it is only necessary to upload the exported thread dump file to quickly view the analysis report without the need to download and install it on the local computer. It is very convenient to use.
- Provides intuitive thread analysis views. It uses graphical displays such as dashboards to make it simple and easy to understand. It also categorizes various thread states, such as blocking, running, timed waiting, waiting, and repetitive stack traces. Through this tool, scalability, performance issues, and availability issues can be quickly and easily resolved.
- Supports REST APIs. FastThread is the first thread dump analysis tool in the industry that supports API calls. With the API interface, we can automate the analysis through scripts or programs, which is suitable for batch operations.
- Supports core dump analysis. Java core dumps contain a lot of information, but the format is very difficult to understand and parse. FastThread can analyze Java core dump files and provide accurate information in a graphical format.
- Analyses hs_err_pid files. Process crashes or fatal errors can cause the JVM to terminate abnormally. At this time, the JVM will generate an hs_err_pid file. This file contains a lot of information, which can be used with FastThread to assist in analysis.
By the way, JVM thread dump is not only available in Java, but also supported by other languages, such as Scala, Jython, JRuby, and so on.
Analyzing thread stack traces online through the FastThread official website is “free”. Now let’s demonstrate the use of this tool through an example program.
Example Program and Thread Dump #
Based on the example code in the previous chapter “JVM Thread Stack Data Analysis”, let’s make some simple modifications to simulate deadlock and thread waiting states.
The example program is as follows:
package demo.jvm0207;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
public class DeadLockSample2 {
public static void main(String[] args) throws Exception {
DeadLockTask deadLockTask = new DeadLockTask();
// Simulate deadlock with multiple threads
new Thread(deadLockTask).start();
new Thread(deadLockTask).start();
// Waiting state
Thread wt = new WaitedThread();
wt.start();
// The current thread waits for another thread to join
wt.join();
}
private static class WaitedThread extends Thread {
@Override
public void run() {
synchronized (DeadLockSample2.class) {
try {
DeadLockSample2.class.wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
// Simple deadlock; locking two objects separately
private static class DeadLockTask implements Runnable {
private Object lockA = new Object();
private Object lockB = new Object();
private AtomicBoolean flag = new AtomicBoolean(false);
public void run() {
try {
if (flag.compareAndSet(false, true)) {
synchronized (lockA) {
TimeUnit.SECONDS.sleep(2);
synchronized (lockB) {
System.out.println("Deadlock inner code");
}
}
} else {
synchronized (lockB) {
TimeUnit.SECONDS.sleep(2);
synchronized (lockA) {
System.out.println("Deadlock inner code");
}
}
}
} catch (Exception e) {
}
}
}
}
}
}
We started the program and found that the system is stuck and not moving.
Then we can use various tools to detect and check the status of the threads. If there are classmates who do not understand, you can refer to the previous chapter "[JVM Thread Stack Data Analysis]".
Thread Dump Snapshot can be used to assist in diagnosing CPU overload, deadlocks, memory exceptions, and long system response times.
So we need to first obtain the corresponding Thread Dump file:
View local JVM process information #
jps -v
Print thread snapshot directly #
jstack -l 51399
Save thread snapshot information to file #
jstack -l 51399 > 51399.thread.dump.txt
The thread dump generated by the jstack tool is roughly as follows:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode):
“Thread-2” #15 prio=5 os_prio=31 tid=0x00007fb3ee805000 nid=0x5a03 in Object.wait() [0x000070000475d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method)
-
waiting on <0x000000076abee388> (a java.lang.Class for demo.jvm0207.DeadLockSample2) at java.lang.Object.wait(Object.java:502) at demo.jvm0207.DeadLockSample2$WaitedThread.run(DeadLockSample2.java:25)
-
locked <0x000000076abee388> (a java.lang.Class for demo.jvm0207.DeadLockSample2)
Locked ownable synchronizers:
-
None
“Thread-1” #14 prio=5 os_prio=31 tid=0x00007fb3ed05d800 nid=0x5903 waiting for monitor entry [0x000070000465a000] java.lang.Thread.State: BLOCKED (on object monitor) at demo.jvm0207.DeadLockSample2$DeadLockTask.run(DeadLockSample2.java:52)
-
waiting to lock <0x000000076abf7338> (a java.lang.Object)
-
locked <0x000000076abf7348> (a java.lang.Object) at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
-
None
“Thread-0” #13 prio=5 os_prio=31 tid=0x00007fb3ef8c1000 nid=0xa703 waiting for monitor entry [0x0000700004557000] java.lang.Thread.State: BLOCKED (on object monitor) at demo.jvm0207.DeadLockSample2$DeadLockTask.run(DeadLockSample2.java:45)
-
waiting to lock <0x000000076abf7348> (a java.lang.Object)
-
locked <0x000000076abf7338> (a java.lang.Object) at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
-
None
“main” #1 prio=5 os_prio=31 tid=0x00007fb3ee006000 nid=0x2603 in Object.wait() [0x0000700002f15000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method)
-
waiting on <0x000000076abf7cf8> (a demo.jvm0207.DeadLockSample2$WaitedThread) at java.lang.Thread.join(Thread.java:1252)
-
locked <0x000000076abf7cf8> (a demo.jvm0207.DeadLockSample2$WaitedThread) at java.lang.Thread.join(Thread.java:1326) at demo.jvm0207.DeadLockSample2.main(DeadLockSample2.java:17)
Locked ownable synchronizers:
-
None
JNI global references: 1358
Found one Java-level deadlock: #
“Thread-1”: waiting to lock monitor 0x00007fb3ee01f698 (object 0x000000076abf7338, a java.lang.Object), which is held by “Thread-0” “Thread-0”: waiting to lock monitor 0x00007fb3ee01f7f8 (object 0x000000076abf7348, a java.lang.Object), which is held by “Thread-1”
Java stack information for the threads listed above: #
“Thread-1”: at demo.jvm0207.DeadLockSample2$DeadLockTask.run(DeadLockSample2.java:52)
- waiting to lock <0x000000076abf7338> (a java.lang.Object)
- locked <0x000000076abf7348> (a java.lang.Object) at java.lang.Thread.run(Thread.java:748) “Thread-0”: at demo.jvm0207.DeadLockSample2$DeadLockTask.run(DeadLockSample2.java:45)
- waiting to lock <0x000000076abf7348> (a java.lang.Object)
- locked <0x000000076abf7338> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
The tool automatically found a deadlock and identified several threads that are in a waiting state. Of course, the information of other threads is omitted above, for example:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode):
"Thread-2" #15 ... in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
"Thread-1" #14 ... waiting for monitor entry
java.lang.Thread.State: BLOCKED (on object monitor)
"Thread-0" #13 ... waiting for monitor entry
java.lang.Thread.State: BLOCKED (on object monitor)
"Service Thread" #12 ... daemon prio=9 ... runnable
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #10 daemon ... waiting on condition
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon ... runnable
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon ... in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
"Reference Handler" #2 daemon ... in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
"main" #1 ... in Object.wait()
java.lang.Thread.State: WAITING (on object monitor)
"VM Thread" ... runnable
"GC task thread#0 (ParallelGC)" ... runnable
"GC task thread#1 (ParallelGC)" ... runnable
"GC task thread#2 (ParallelGC)" ... runnable
"GC task thread#3 (ParallelGC)" ... runnable
"GC task thread#4 (ParallelGC)" ... runnable
"GC task thread#5 (ParallelGC)" ... runnable
"GC task thread#6 (ParallelGC)" ... runnable
"GC task thread#7 (ParallelGC)" ... runnable
"VM Periodic Task Thread" ... waiting on condition
After obtaining the thread snapshot information, let's take a look at how to use the FastThread analysis tool.
### FastThread Usage Example
Open the official homepage: <http://fastthread.io/>.
#### **File Upload Method**
Select a file and upload it, then click the "Analyze" button.
#### **Text Upload Method**
The steps are similar for both methods. Choose the RAW option to upload the text string, and then click the "Analyze" button.
#### **Analysis Result Page**
Wait a moment and you will be automatically redirected to the analysis result page.
Here you can see basic information and some links on the right side:
* Share Report: You can easily send the report to other teammates.
#### **Thread Count Summary**
Scroll down the page to see the thread count summary report.
From this report, you can visually see that there are a total of 26 threads, including 19 running threads, 5 waiting threads, and 2 blocked threads.
A pie chart on the right side also shows the proportion of each state.
#### **Thread Group Analysis**
The threads are automatically grouped by name next.
Here you can see the benefit of naming threads! If we name our thread pool uniformly, we can easily understand the usage of related resource pools.
> So when using a thread pool in the code, it is a good habit to add a unified thread name!
#### Analysis of Daemon Threads
Next is the analysis of daemon threads:
Here you can see the statistics of daemon threads and foreground threads.
#### Deadlock Detection
Of course, there is also deadlock analysis:
You can see that the deadlock detection results given by various tools are similar. It is not difficult to analyze, which includes the thread name, method call stack information, and the lock being waited for.
#### Thread Invocation Stack
And the thread invocation situation:
After that, the details of these threads are shown:
This information simply displays the relevant method call stack.
#### Hot Methods Statistics
Hot methods are a key point to pay attention to. The more they are called, the more likely this area may be a bottleneck in the system's performance.
Here, the methods currently executing in this snapshot are displayed. If you only look at the sampling of hot methods, the more accurate tool is hprof, which is built into the JDK.
However, if there are many methods blocked or waiting, the location of hot methods shown in the thread snapshot can quickly determine the code line where the problem occurs.
#### CPU Consumption Information
The hint here is not very obvious, but it provides some learning resources. Please refer to the blog link addresses given at the end of this article.
#### GC Thread Information
Here, you can see that there are 8 GC threads, which is normal considering the specific number of CPU cores.
If there are too many or too few GC threads, it can cause many problems, which we will explain with examples in later chapters.
#### Thread Stack Depth
Here, they are all less than 10, which means the stacks are not deep.
#### Complex Deadlock Detection
Next are the complex deadlock detection and Finalizer thread information.
Simple deadlock refers to two threads waiting for each other's resource locks. Then what is a complex deadlock? This question is left for you to search for.
#### Flame Graph
The flame graph is quite interesting, summarizing all thread call stacks into one image.
#### Call Stack Tree
What if we merge all the call stacks together and look at them as a whole?
Tree structure is also useful in some cases, such as when a large number of threads are executing similar call stack paths.
All of the above information helps us analyze and troubleshoot JVM problems. The advantage of graphical tools over command-line tools is that they are intuitive, convenient, and fast, saving us time filtering unnecessary interference information.
### Reference Links
* [8 Ways to Capture Java Thread Dumps](https://www.jianshu.com/p/433dfb0f7879) (Chinese content)
* [Thread Dump Options](https://blog.fastthread.io/2016/06/06/how-to-take-thread-dumps-7-options/)
* [FastThread Official Blog](https://blog.fastthread.io/)