28 'Testnginx' Can Also Be Used Like This

28 ’testnginx’ Can Also Be Used Like This #

Hello, I am Wen Ming.

In the previous two sections, you have already mastered most of the usage of test::nginx. I believe you are now able to understand most of the test cases in the OpenResty project. This should be sufficient for learning OpenResty and its surrounding libraries.

However, if you aspire to become a contributor to OpenResty’s code, or if you are using test::nginx to write test cases in your own project, then you need to learn some more advanced and complex usages.

Today’s content may be the most “advanced” part of this column, as it has never been shared before. Taking the lua-nginx-module, which is the core module of OpenResty, as an example, there are a total of over 70 contributors worldwide, but not every contributor has written test cases. So, if you complete today’s lesson, your understanding of test::nginx will definitely be among the top 100 globally.

Debugging in Testing #

First, let’s take a look at several essential primitives that developers often use for debugging in testing. We will introduce the usage scenarios of these debugging-related primitives one by one.

ONLY #

Many times, we add a new test case to the existing test case set. If the test file contains a lot of test cases, it is obviously time-consuming to run them from start to finish. This is particularly noticeable when you need to repeatedly modify test cases.

So, is there any way to only run a specific test case that you specify? The ONLY marker can easily achieve this:

=== TEST 1: sanity
=== TEST 2: get
--- ONLY

The above pseudocode shows how to use this primitive. Put --- ONLY at the end of the test case that you want to run separately. When you use the prove command to run this test case file, it will ignore all other test cases and only run this one.

However, this is only suitable for debugging purposes. Therefore, when the prove command finds the ONLY marker, it will also give a prompt to remind you to remove it before submitting the code.

SKIP #

The corresponding requirement for executing only one test case is to ignore a specific test case. The SKIP marker is generally used for testing functionality that has not been implemented yet:

=== TEST 1: sanity
=== TEST 2: get
--- SKIP

From this pseudocode, it can be seen that its usage is similar to ONLY. As we are practicing test-driven development, we need to write test cases first. However, during collective coding implementation, it may be necessary to delay the implementation of a particular functionality due to implementation difficulty or priority. In this case, you can skip the corresponding test case set and remove the SKIP marker when the implementation is complete.

LAST #

There is also a commonly used marker called LAST. Its usage is also straightforward - all the test case sets before it will be executed, and the ones after it will be ignored:

=== TEST 1: sanity
=== TEST 2: get
--- LAST
=== TEST 3: set

You might wonder, I understand ONLY and SKIP, but what is the use of LAST? In fact, sometimes your test cases have dependencies, and the tests after executing the previous test cases are meaningful. In such cases, LAST is very useful for debugging.

Test Plan #

Among all the primitives in test::nginx, plan is the most frustrating and difficult to understand. It originates from the Test::Plan module in Perl, so the documentation is not in test::nginx. Finding an explanation of it is not easy, so I am placing it at the beginning to introduce it. I have seen several contributors to OpenResty’s code stumble into this pitfall and struggle to get out.

Below is an example that you can find in the beginning of each file in the official OpenResty test suite:

plan tests => repeat_each() * (3 * blocks());

Here, the meaning of plan is how many checks should be done in the entire test file. If the actual result does not match the plan, the entire test fails.

In this example, if the value of repeat_each is 2 and there are a total of 10 test cases, then the value of the plan should be 2 x 3 x 10 = 60. The only thing you might not understand here is the meaning of the number 3, which seems like a magic number!

Don’t worry, let’s continue with the example, and you will understand it soon. But first, can you accurately calculate the correct value of the plan in the following test case?

=== TEST 1: sanity
--- config
    location /t {
        content_by_lua_block {
            ngx.say("hello")
        }
    }
--- request
GET /t
--- response_body
hello

I believe everyone would come to the conclusion that the plan is 1 because only the response_body is being checked.

However, that is not the case! The correct answer is that the plan is 2. Why? Because test::nginx implicitly includes a check, which is --- error_code: 200, and it verifies whether the HTTP response code is 200 by default.

So, the true meaning behind the magic number 3 is that each test is explicitly checked twice, for example, the body and error log, while implicitly checking the response code.

Since this aspect is prone to mistakes, my recommendation is to directly disable the plan using the following method:

use Test::Nginx::Socket 'no_plan';

If it cannot be disabled, for example, when encountering an inaccurate plan in the official OpenResty test suite, it is suggested not to delve into the reasons and simply add or subtract numbers in the plan expression:

plan tests => repeat_each() * (3 * blocks()) + 2;

This is also a method used by the official team.

Preprocessor #

We know that there may be some common settings between different test cases in the same test file. If you repeat the settings in each test case, the code will appear redundant and it will be cumbersome to modify later.

In this case, you can use the add_block_preprocessor directive to add a piece of Perl code, for example, like this:

add_block_preprocessor(sub {
    my $block = shift;

    if (!defined $block->config) {
        $block->set_value("config", <<'_END_');
    location = /t {
        echo $arg_a;
    }
    _END_
    }
});

This preprocessor will add a configuration section for all test cases, and the content inside is location /t. This way, you can omit the config in your subsequent test cases and access it directly:

=== TEST 1:
--- request
    GET /t?a=3
--- response_body
3

=== TEST 2:
--- request
    GET /t?a=blah
--- response_body
blah

Custom Functions #

In addition to adding Perl code in the preprocessor, you can also add custom functions freely before the run_tests primitive, which are referred to as custom functions.

Here is an example that adds a function for reading files and combines it with the eval directive to implement the functionality of uploading files via POST:

sub read_file {
    my $infile = shift;
    open my $in, $infile
        or die "cannot open $infile for reading: $!";
    my $content = do { local $/; <$in> };
    close $in;
    $content;
}

our $CONTENT = read_file("t/test.jpg");

run_tests;

__DATA__

=== TEST 1: sanity
--- request eval
"POST /\n$::CONTENT"

Random Execution #

In addition to the aforementioned points, test::nginx has another little-known pitfall: by default, it executes test cases in random order, rather than in the order specified by the test cases’ sequencing and numbering.

The intention behind this design is to uncover more issues. After all, for every test case, the Nginx process is closed and a new one is started for execution, so the results should not be dependent on the order.

This indeed holds true for lower-level projects. However, for application-level projects that involve external databases or other persistent storage, the random execution can lead to incorrect results. Since each execution is random, errors may or may not be reported, and even if reported, they may be different each time. Clearly, this can be confusing for developers, and I myself have stumbled upon it many times.

Therefore, my advice is to disable this feature. You can use the following two lines of code to do so:

no_shuffle();
run_tests;

The no_shuffle primitive is used to disable the randomness and ensure that the tests are executed strictly in the order specified by the test cases.

reindex #

Finally, let’s talk about a less brain-burning and more relaxed topic. OpenResty’s test case set has strict formatting requirements. There should be 3 line breaks between each test case, and the serial numbers of the test cases must strictly increase.

Fortunately, we have the corresponding automation tool called reindex to handle these tedious tasks. It is hidden in the [openresty-devel-utils] project. Since there is no documentation to introduce it, few people know about it.

Interested students can try to shuffle the serial numbers of the test cases or add/remove the number of line breaks between them, and then use this tool to organize them and see if they can be restored.

Conclusion #

And that’s the end of the introduction to test::nginx. Of course, it has many more features, but we’ve only covered the most essential ones. Rather than giving you fish, I’ve taught you the basic methods and points to note when testing. The rest is up to you to explore in the official test case collection.

Before we finish, I’ll leave you with a question. Do you have testing in your project development? And what framework do you use for testing? I welcome your feedback and discussion on this question. Feel free to share this article with others to encourage more exchange and learning.