24 Basic Rules and Processes of Testing - Part 2 #

Hello, I’m Haolin. Today I will continue to share the topic of basic rules and procedures for testing.

Go language is a programming language that emphasizes program testing. Therefore, in the previous article, I emphasized the importance of program testing and introduced the basic rules and main procedures for the go test command. Today, we will continue to share the basic rules and procedures for testing. This article contains more code and instructions. You can click on the article to view the original content.

Knowledge Expansion #

Question 1: How to interpret the test results of functional testing? #

Let’s take a look at the following test command and its result:

$ go test puzzlers/article20/q2
ok   puzzlers/article20/q2 0.008s

The line starting with $ indicates the command I entered. Here, I entered go test puzzlers/article20/q2, which means I want to test the code package with the import path puzzlers/article20/q2. The line below the command is the brief result of this test.

This brief result consists of three parts. The leftmost ok indicates that the test was successful, meaning no unexpected test results were found.

Of course, this is determined entirely by the test code we wrote. We always assume that the test code itself has no bugs and faithfully implements our testing intentions. In the middle of the test result, the import path of the tested code package is displayed.

On the rightmost side, the time it took for this test on the code package is displayed. Here, it shows 0.008s, which means 8 milliseconds. However, when we immediately run the command for the second time, the output of the test result will be slightly different, as shown below:

$ go test puzzlers/article20/q2
ok   puzzlers/article20/q2 (cached)

As you can see, the rightmost part of the result is no longer the test execution time, but (cached). This indicates that since both the test code and the tested code have not changed, the go test command simply prints out the previously cached result of the successful test.

The go command usually caches the results of program builds for reuse in future builds. You can view the path of the cache directory by running the command go env GOCACHE. The cached data always accurately reflects the real situation of the relevant source code files, build environments, compiler options, etc., at the time.

Once there are any changes, the cached data becomes invalid and the go command will perform the actions again. Therefore, we don’t need to worry that the printed cache data is not the real-time result. The go command periodically deletes unused cache data, but if you want to manually delete all cache data, simply run the command go clean -cache.

For test results that pass, the go command also caches them. Running go clean -testcache will delete all cached test results. However, this will certainly not delete any cached build results.

In addition, setting the value of the environment variable GODEBUG can slightly change the caching behavior of the go command. For example, setting the value to gocacheverify=1 will cause the go command to bypass any cached data and actually perform the actions again, and then check if the new results are consistent with the existing cache data.

In summary, we don’t need to worry about the existence of cached data because they will not interfere with the go test command printing the correct test results.

You may ask, what will the result printed by the command be if the test fails? If the only parameter of the functional test function is named t, and we call the t.Fail method within it, although the current test function will continue to execute, the result will indicate that the test has failed. As shown below:

$ go test puzzlers/article20/q2
--- FAIL: TestFail (0.00s)
 demo53_test.go:49: Failed.
FAIL
FAIL puzzlers/article20/q2 0.007s

The command we ran is the same as before, but I added a new functional test function TestFail and called t.Fail inside it. The test result shows that the test on the tested code package has failed due to the failure of the TestFail function.

Note that for failed tests, the go test command does not cache the results. Therefore, each test in this case will produce a brand new result. Additionally, if a test fails, the go test command will cause the regular test log in the failed test function to be printed as well.

In the test result here, the reason why the line “demo53_test.go:49: Failed.” is displayed is because I wrote the code t.Log("Failed.") below the call expression t.Fail() in the TestFail function.

The purpose of the t.Log method and t.Logf method is to print regular test logs. However, when the test is successful, the go test command will not print such logs. If you want to see all regular test logs in the test result, you can add the -v flag when running the go test command.

If we want a test function to immediately fail during execution, we can call the t.FailNow method in that function.

I will change t.Fail() in the TestFail function to t.FailNow() below.

Unlike t.Fail(), after t.FailNow() is executed, the current function will immediately stop executing. In other words, all the code after that line of code will lose the chance to execute. After making this modification, I run the command again with the same result as before:

--- FAIL: TestFail (0.00s)
FAIL
FAIL puzzlers/article20/q2 0.008s

Obviously, the regular test logs that were displayed in the results before did not appear here.

By the way, if you want to print the log of a failed test while the test is failing, you can directly call the t.Error or t.Errorf methods.

The former is equivalent to a continuous call to t.Log and t.Fail, while the latter is similar but equivalent to calling t.Logf first.

In addition, there are also t.Fatal and t.Fatalf methods, which will immediately terminate the execution of the current test function and declare it as a failed test after printing the failure error log. Specifically, they both call t.FailNow in the end.

Alright, now you should be able to understand the test results of the functional tests, right?

Question 2: How to interpret the test results of performance tests? #

Performance test results have many similarities with functional test results. Here, we only focus on the differences of the former. Take a look at the print result below:

$ go test -bench=. -run=^$ puzzlers/article20/q3
goos: darwin
goarch: amd64
pkg: puzzlers/article20/q3
BenchmarkGetPrimes-8      500000       2314 ns/op
PASS
ok   puzzlers/article20/q3 1.192s

When running the go test command, I added two flags. The first flag and its value -bench=., only performs performance tests when this flag is present. The value . of this flag means execute any performance test function, of course, the function name still needs to comply with the basic rules of Go program testing.

The second flag and its value -run=^$ indicate which functional test functions need to be executed, also based on the function name. The value ^$ of this flag means: only execute functional test functions with empty names, in other words, do not execute any functional test functions.

You may have noticed that both values of these two flags are regular expressions. In fact, they can only accept regular expressions as values. In addition, if the go test command is run without the -run flag, it will execute all functional test functions in the tested code package.

Now let’s look at the test results, and focus on the content of the third to last line. BenchmarkGetPrimes-8 is the name of a single performance test. It represents that the command executed the performance test function BenchmarkGetPrimes, and the maximum P count at that time was 8.

The maximum P count represents the maximum number of logical CPUs that can run goroutines at the same time. Here, logical CPU can also be referred to as CPU core, but it is not exactly equivalent to the actual CPU core in the computer. It is just a concept inside the Go runtime system, representing its ability to run goroutines concurrently.

By the way, the number of CPU cores on a computer means how many program instructions it can execute at the same time, representing its ability to parallel process program instructions.

We can change the maximum P count by calling the runtime.GOMAXPROCS function, or by adding the -cpu flag when running the go test command to set up a list of maximum P counts for the command to use in multiple tests.

As for how to use this flag, and how the go test command changes its test flow based on it, we will discuss it in the next article.

On the right side of the performance test name is the actual number of times the tested function (GetPrimes function) was executed when the go test command last executed the performance test function (BenchmarkGetPrimes function). What does this mean?

When executing the performance test function, the go test command gives it a positive integer. If the unique parameter of the test function is named b, this positive integer is represented by b.N. We should write code in the test function accordingly, for example:

for i := 0; i < b.N; i++ {
    GetPrimes(1000)
}

I called the GetPrimes function in a loop that iterates b.N times, and gave it a parameter value of 1000. The go test command will first try to set b.N to 1, and then execute the test function.

If the execution time of the test function does not exceed the limit, which is 1 second by default, the command will increase the value of b.N and execute the test function again, and so on, until the time is greater than or equal to the limit.

When the time of a certain execution is greater than or equal to the limit, we say that it is the last execution of this test function by the command. The value of b.N at that time will be included in the test result, which is the 500000 in the test result above.

We can simply call this value the number of executions, but it is important to note that it refers to the number of executions of the tested function, not the number of executions of the performance test function.

Finally, let’s look at the right side of this number of executions. 2314 ns/op means that the average time for a single execution of the GetPrimes function is 2314 nanoseconds. This is calculated by dividing the execution time of the last execution of the test function by the number of executions of the tested function.

(The basic interpretation of performance test results)

These are the basic interpretations of the default performance test results. Do you understand them?

Summary #

Note that there are some differences in the way command execution test flows are carried out for functional testing and performance testing. Another important issue is how we interpret the information provided by the go test command when interacting with it. Only with correct interpretation can you know the success or failure of the tests, the specific reasons for failure, and the severity, and so on.

In addition, for performance testing, you also need to pay attention to the resource usage prompts and various performance metrics output by the command.

In these two articles, we have learned a lot together, but it’s actually not enough. We’ve only discussed the basic usage of the go test command and the testing package.

In the next article, we will also discuss more advanced topics. This will involve various flags of the go test command, more APIs of the testing package, and more complex test results.

Thought Question #

When writing example test functions, how do we specify the expected print content?

Click here to view the detailed code for the Go Language column article.