31 Pdb C Profile for Debugging and Performance Analysis

31 pdb - cProfile for Debugging and Performance Analysis #

Hello, I am Jingxiao.

In a real production environment, debugging and performance profiling of code are topics that can never be avoided. The main scenarios for debugging and performance profiling typically include:

First, when there are problems with the code itself, we need to identify the root cause and fix it.
Second, when there are efficiency issues with the code, such as excessive resource consumption or increased latency, we need to debug it.
Third, when developing new features, testing is generally required.

When encountering these scenarios, what tools should we use, how to use these tools correctly, and what steps should we follow, are the topics we will discuss in this lesson.

Debugging Code Using pdb #

The Necessity of pdb #

First, let’s talk about debugging code. Some may wonder: debugging? Isn’t that just using print() statements in the program?

Yes, you’re right. Using print() statements in the program is indeed a common way to debug, especially for small programs. Because you can print out the variable values by adding print() statements in the corresponding places. If the program is small and runs quickly, it is convenient to use print().

However, when it comes to large programs, the cost of debugging a single run is high. Especially for some tricky examples, they usually require repeated debugging and tracing the context code to find the root cause of the error. In this case, relying solely on printing is not very efficient.

Imagine this scenario: your most commonly used Geektime App has recently encountered a bug, and some users are unable to log in. So, the backend engineers start debugging.

They suspect that the error lies in a few specific functions. If they use print() statements to debug, a possible scenario could be that the engineers add print() statements in what they think are the 10 most likely places where the bug might occur. Then they run the entire function block (which takes 5 minutes from start to finish) and check if the printed values match their expectations.

If the values match the expectations and they can directly find the root cause of the error, it would be the best outcome. But in reality,

Either the values don’t match the expectations, and they need to repeat the above steps to continue debugging.
Or even if the values match the expectations, the previous operations only narrow down the range of the error code, so they still need to add print() statements and run the code module again (another 5 minutes) for debugging.

As you can see, this efficiency is very low. Even if you encounter slightly more complex cases, two or three engineers may spend an entire afternoon on them.

Some may say, don’t many IDEs have built-in debugging tools now?

This is also true. For example, Pycharm, which we commonly use, allows us to easily set breakpoints in the program. This way, when the program reaches a breakpoint, it will automatically stop and you can easily view the values of variables in the environment and execute corresponding statements, greatly improving the efficiency of debugging.

Seeing this, you may wonder why we still need to learn pdb if the problem is already solved. In fact, in many large companies, the creation and iteration of products often require support from multiple programming languages. Moreover, the company may also develop many internal interfaces to try to combine as many languages as possible.

This means that in many cases, IDEs for individual languages do not support UI-style breakpoint debugging of mixed code or only support certain functional modules. Additionally, considering that a lot of code has been moved to notebooks like Jupyter, developers are often required to use the command line to debug the code.

And Python’s pdb is exactly its built-in debugging library. It provides interactive source code debugging functionality for Python programs and is a command-line version of the IDE breakpoint debugger, perfectly solving the problem we just discussed.

How to Use pdb #

Having understood the importance and necessity of pdb, let’s now see how pdb should be used in Python.

To start pdb debugging, all we need to do is add the following two lines of code in the program: import pdb and pdb.set_trace(). For example, consider the following simple example:

a = 1
b = 2
import pdb
pdb.set_trace()
c = 3
print(a + b + c)

When we run this program, the output interface will look like this, indicating that the program has reached the line pdb.set_trace() and paused, waiting for user input.

> /Users/jingxiao/test.py(5)<module>()
-> c = 3

At this point, we can execute all the operations that can be performed in an IDE breakpoint debugger, such as printing. The syntax for printing is p <expression>:

(pdb) p a
1
(pdb) p b
2

As you can see, I printed the values of a and b, which are 1 and 2, respectively, matching the expectations. Why didn’t I print c? Clearly, printing c would throw an exception because the program has only executed a few lines so far, and the variable c has not been defined:

(pdb) p c
*** NameError: name 'c' is not defined

In addition to printing, there is also a commonly used operation n which means to continue executing the code to the next line. The usage is as follows:

(pdb) n
-> print(a + b + c)

And the command "l" lists the 11 lines of source code above and below the current line of code, allowing developers to familiarize themselves with the code surrounding the breakpoint:

(pdb) l
  1  	a = 1
  2  	b = 2
  3  	import pdb
  4  	pdb.set_trace()
  5  ->	c = 3
  6  	print(a + b + c)

The command "s" stands for step into, which means stepping into the corresponding code. At this point, the command line will display the word "--Call--", and when you finish executing the code block inside, the command line will display the word "--Return--".

Let’s look at the following example:

def func():
    print('enter func()')

a = 1
b = 2
import pdb
pdb.set_trace()
func()
c = 3
print(a + b + c)

> /Users/jingxiao/test.py(9)<module>()
-> func()
(Pdb) s
--Call--
> /Users/jingxiao/test.py(1)func()
-> def func():
(Pdb) l
  1  ->	def func():
  2  		print('enter func()')
  3
  4
  5  	a = 1
  6  	b = 2
  7  	import pdb
  8  	pdb.set_trace()
  9  	func()
 10  	c = 3
 11  	print(a + b + c)

(Pdb) n
> /Users/jingxiao/test.py(2)func()
-> print('enter func()')
(Pdb) n
enter func()
--Return--
> /Users/jingxiao/test.py(2)func()->None
-> print('enter func()')

(Pdb) n
> /Users/jingxiao/test.py(10)<module>()
-> c = 3

Here, we used the command "s" to enter the inner part of the function func(), which displays "--Call--". And when we finish executing the statements inside the func() function and return, it displays "--Return--".

In addition:

The corresponding command "r" represents step out, which means continuing execution until the current function completes and returns.
The command "b [ ([filename:]lineno | function) [, condition] ]" can be used to set a breakpoint. For example, if I want to add a breakpoint on line 10 of the code, I can enter "b 11" in pdb mode.
The command "c" means continuously execute the program until the next breakpoint is encountered.

Of course, in addition to these common commands, there are many other commands available, which I will not go into detail here. You can refer to the corresponding official documentation (https://docs.python.org/3/library/pdb.html#module-pdb) to familiarize yourself with these usages.

Profiling with cProfile #

Regarding debugging, I have discussed enough for now. In fact, besides debugging, performance profiling is also a necessary skill for every developer.

In our daily work, we often encounter problems like this: in production, I find that a certain function module of the product is inefficient, has high latency, and consumes a lot of resources, but I don’t know where the problem is.

At this point, it is extremely important to profile the code.

The so-called profiling here refers to the dynamic analysis of each part of the code, such as accurately calculating the time consumed by each module. This way, you can identify the bottlenecks in the program and make modifications or optimizations accordingly. Of course, this does not require much effort. In Python, these needs can be fulfilled using cProfile.

Let’s take an example. Suppose I want to calculate the Fibonacci sequence. Using recursion, we can easily write the following code:

def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)

def fib_seq(n):
    res = []
    if n > 0:
        res.extend(fib_seq(n-1))
    res.append(fib(n))
    return res

fib_seq(30)

Next, I want to test the overall efficiency of this code and the efficiency of each part. To do this, all I need to do is import the cProfile module at the beginning and run cProfile.run() at the end:

import cProfile
cProfile.run('fib_seq(30)')

Alternatively, a simpler way is to include the option “-m cProfile” in the command to run the script:

python3 -m cProfile xxx.py

After the program finishes running, we can see the following output:

There are some parameters here that you may not be familiar with, and I will briefly introduce them:

ncalls: It indicates the number of times the corresponding code/function is called.
tottime: It indicates the total time required for the corresponding code/function to execute (Note that it does not include the execution time of other code/functions called by it).
tottime percall: This is the result of dividing tottime by ncalls, i.e., tottime / ncalls.
cumtime: It indicates the total time required for the corresponding code/function to execute, including the execution time of other code/functions called by it.
cumtime percall: This is the average result of dividing cumtime by ncalls.

After understanding these parameters, let’s take a look at the output. We can clearly see that the bottleneck in the efficiency of this program is the second line where the function fib() is called more than 7 million times.

Is there any way to improve it? The answer is yes. Through observation, we found that many calls to fib() in the program are actually redundant. Therefore, we can use a dictionary to store the calculated results to prevent duplication. The improved code is shown below:

def memoize(f):
    memo = {}
    def helper(x):
        if x not in memo:            
            memo[x] = f(x)
        return memo[x]
    return helper

@memoize
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)


def fib_seq(n):
    res = []
    if n > 0:
        res.extend(fib_seq(n-1))
    res.append(fib(n))
    return res

fib_seq(30)

Now, if we profile it again, we will get a new output that clearly shows a significant improvement in efficiency.

This simple example demonstrates the basic usage of cProfile, which is also the main point I wanted to discuss today. Of course, cProfile has many other functions and can be used in conjunction with the stats class. You can read the corresponding official documentation to learn more.

Summary #

In this lesson, we have learned about the commonly used debugging tool pdb and the classic performance analysis tool cProfile in Python. pdb provides a universal and interactive solution for efficient debugging of Python programs, while cProfile provides detailed analysis of the execution efficiency of each code block, helping us optimize and improve our programs.

For more usage information, you can practice with them through their official documentation. They are not difficult and practice makes perfect.

Thought-provoking Question #

Finally, let’s leave an open-ended question for discussion. What are the debugging and performance analysis tools you often use in your work? Have you discovered any unique tips for their usage? Have you ever used tools like pdb, cProfile, or others?

Feel free to leave your comments below and discuss with me. It’s also welcome if you share this article. Let’s communicate and improve together.