27 'Testnginx' a Comprehensive Test Methodology

27 ’testnginx’- A Comprehensive Test Methodology #

Hello, I am Wen Ming.

After studying the previous lesson, you have gained a preliminary understanding of test::nginx and have run the simplest example. However, in actual open source projects, the test cases written by test::nginx are obviously much more complex and difficult to grasp than the example code. Otherwise, it would not be such a roadblock.

In this lesson, I will familiarize you with the instructions and testing methods frequently used in test::nginx, with the aim of enabling you to understand most of the test case collections in the OpenResty project and have the ability to write more realistic test cases. Even if you have not contributed code to OpenResty yet, being familiar with the testing framework of OpenResty will still provide you with a lot of inspiration for designing and writing test cases in your daily work.

In essence, the testing in test::nginx is based on the configuration of each test case. First, it generates nginx.conf and starts an Nginx process. Then, it simulates a client to initiate a request, including the specified request body and headers. Next, the Lua code in the test case handles the request and generates a response. At this point, test::nginx parses the response body, headers, error logs, and other key information, and compares them with the test configuration. If any discrepancies are found, an error is reported, and the test fails; otherwise, it is considered successful.

test::nginx provides many original primitives in the DSL. I have categorized them according to the flow of Nginx configuration, sending requests, handling responses, and checking logs. These 20% of the features can cover 80% of the use cases, so you must grasp them firmly. As for the more advanced primitives and usage methods, we will introduce them in the next lesson.

Nginx Configuration #

Let’s first take a look at Nginx configuration. In the test::nginx primitive, any configuration related to Nginx will have the keyword config, such as config, stream_config, http_config, etc., as mentioned in the previous section.

Their function is the same, which is to insert specified Nginx configuration in different Nginx contexts. These configurations can be Nginx directives or Lua code encapsulated in content_by_lua_block.

During unit testing, config is the most commonly used primitive. We use it to load Lua libraries and call functions for white-box testing. Below is an excerpt from a test code, which cannot run completely. It comes from a real open-source project. If you are interested, you can click on the link to view the complete test or try running it locally.

=== TEST 1: sanity
--- config
    location /t {
        content_by_lua_block {
            local plugin = require("apisix.plugins.key-auth")
            local ok, err = plugin.check_schema({key = 'test-key'})
            if not ok then
                ngx.say(err)
            end
            ngx.say("done")
        }
    }

The purpose of this test case is to test if the function check_schema in the code file plugins.key-auth can work properly. It uses the Nginx directive content_by_lua_block in location /t to require the module that needs to be tested and directly call the function to be checked.

This is a common approach to white-box testing in test::nginx. However, this configuration alone cannot complete the testing. Now let’s continue to see how to make client requests.

Sending Requests #

Simulating client requests involves a lot of details, so let’s start with the simplest example of sending a single request.

request #

Continuing from the previous test case, if you want your unit test code to be executed, you need to make an HTTP request to the address specified in the config as /t, as shown in the following test code:

--- request
GET /t

In this code, the request primitive is used to send a GET request to the /t address. We did not specify the IP address, domain, or port to access, nor did we specify whether it’s HTTP 1.0 or HTTP 1.1. test::nginx takes care of these details for you, so you don’t need to worry about them. This is one of the advantages of DSL - you only need to focus on the business logic without being bothered by various details.

At the same time, this also provides some flexibility. By default, the protocol is HTTP 1.1. If you want to test HTTP 1.0, you can specify it separately:

--- request
GET /t  HTTP/1.0

In addition to the GET method, the POST method also needs to be supported. In the following example, you can POST the string hello world to the specified address:

--- request
POST /t
hello world

Similarly, test::nginx automatically calculates the length of the request body and adds the host and connection request headers to ensure that it is a normal request.

Of course, for the sake of readability, you can add comments in it. Anything starting with # will be recognized as a code comment:

--- request
# post request
POST /t
hello world

request also supports more complex and flexible patterns, which is to embed Perl code directly with the eval filter, as test::nginx is written in Perl. This approach is like opening a backdoor outside the DSL. If the current DSL primitives cannot meet your needs, then the eval method, which directly executes Perl code, can be said to be the “ultimate weapon”.

Regarding the usage of eval, let’s start with a few simple examples. We will introduce more complex ones in the next lesson:

--- request eval
"POST /t
hello\x00\x01\x02
world\x03\x04\xff"

In the first example, we use eval to specify unprintable characters, which is one of its uses. The content between double quotes is treated as a Perl string and then passed to request as an argument.

Here’s a more interesting example:

--- request eval
"POST /t\n" . "a" x 1024

However, to understand this example, you need to understand some Perl string knowledge. Here’s a quick summary:

  • In Perl, a dot is used to represent string concatenation, which is similar to Lua’s two dots.
  • Lowercase x is used to indicate the repetition of characters. For example, "a" x 1024 means the character a is repeated 1024 times.

So, the meaning of the second example is to send a POST request to the /t address with 1024 characters a in the request.

pipelined_requests #

After understanding how to send a single request, let’s take a look at how to send multiple requests. In test::nginx, you can use the pipelined_requests primitive to send multiple requests in the same keep-alive connection:

--- pipelined_requests eval
["GET /hello", "GET /world", "GET /foo", "GET /bar"]

For example, this example will sequentially access these 4 interfaces in the same connection. This has two benefits:

  • First, it simplifies the test code and compresses 4 test cases into one.
  • Second and most importantly, you can use pipelined requests to detect if there are any exceptions in the code logic when accessed multiple times.

You may wonder that I’m writing multiple test cases one by one, so when executed, the code will also be executed multiple times, thus covering the second problem mentioned above?

In fact, this involves the execution mode of test::nginx. It does not work as you imagined. In fact, after each test case is executed, test::nginx closes the current Nginx process, and naturally, all the data in memory disappears. When running the next test case, it will regenerate nginx.conf and start a new Nginx worker. This mechanism ensures that the test cases do not affect each other.

Therefore, when you want to test multiple requests, you need to use the pipelined_requests primitive. Based on it, you can simulate various scenarios such as rate limiting, bandwidth limiting, and concurrent limiting to test if your system is functioning properly. We will continue to discuss this in the next lesson because it involves the combination of multiple instructions and primitives.

repeat_each #

We mentioned earlier that when testing multiple requests in a test case, how to run the same test multiple times?

For this question, test::nginx provides a global setting: repeat_each. It is actually a Perl function, and by default it is repeat_each(1), which means that the test case is only executed once. So far, we have not set it separately in the previous test cases.

Naturally, you can set it before the run_test() function, for example, change the parameter to 2:

repeat_each(2);
run_tests();

Then each test case will be executed twice, and so on.

more_headers #

After discussing the request body, let’s take a look at the request headers. As mentioned above, test::nginx will include the host and connection headers by default when sending requests. So how do you set other request headers?

Actually, more_headers is designed for this purpose:

--- more_headers
X-Foo: blah

You can use it to set various custom headers. If you want to set multiple headers, simply set multiple lines:

--- more_headers
X-Foo: 3
User-Agent: openresty

Handling Response #

After sending the request, the most important part in test::nginx of test is handling the response. Here, we will judge whether the response meets the expectations. We will introduce four parts in order: response body, response headers, error code, and error log.

response_body #

The counterpart of the request primitive is response_body. Below is an example of how they are used in the configuration:

=== TEST 1: sanity
--- config
    location /t {
        content_by_lua_block {
            ngx.say("hello")
        }
    }
--- request
GET /t
--- response_body
hello

In this test case, the response body is successfully matched with the expected body, which is hello. Otherwise, an error will be reported. But what if the body is very long? How can we test it appropriately? Don’t worry, test::nginx has already considered this for you. It supports using regular expressions to match the response body. For example, you can write:

--- response_body_like
^he\w+$

This way, you can perform flexible checks on the response body. Not only that, test::nginx also supports the unlike operation:

--- response_body_unlike
^he\w+$

In this case, if the response body is hello, the test will fail.

Following the same logic, after understanding the testing of individual requests, let’s take a look at the testing of multiple requests. Here is an example using pipelined_requests:

--- pipelined_requests eval
["GET /hello", "GET /world", "GET /foo", "GET /bar"]
--- response_body eval
["hello", "world", "oo", "bar"]

Of course, it is important to note that you need to have one response corresponding to each request you send.

response_headers #

The second part is about response headers. Response headers are similar to request headers, and each line corresponds to a key-value pair of headers:

--- response_headers
X-RateLimit-Limit: 2
X-RateLimit-Remaining: 1

Similar to testing the response body, response headers also support regular expressions and the unlike operation. They are response_headers_like, raw_response_headers_like, and raw_response_headers_unlike.

error_code #

Next, let’s talk about the error code. The testing of error codes supports direct comparison and the like operation. For example, here are two examples:

--- error_code: 302

--- error_code_like: ^(?:500)?$

When there are multiple requests, the error_code naturally needs to be tested multiple times:

--- pipelined_requests eval
["GET /hello", "GET /hello", "GET /hello", "GET /hello"]
--- error_code eval
[200, 200, 503, 503]

error_log #

The last item to be tested is the error log. In most test cases, there are no error logs generated. We can use no_error_log to check for this:

--- no_error_log
[error]

In the above example, if the string [error] appears in Nginx’s error log error.log, the test will fail. This is a very common feature, and it is recommended to include error log checks in your normal tests.

On the other hand, we also need to write many exceptional test cases to verify whether our code handles errors correctly. In this case, we need the error log to contain specific strings, and this is where the error_log comes into play:

--- error_log
hello world

The above configuration actually checks if hello world appears in the error.log. Of course, you can embed regular expressions using eval, for example:

--- error_log eval
qr/\[notice\] .*?  \d+ hello world/

Final Thoughts #

Today, we learned how to send requests and check responses in test::nginx, including the body, headers, response code, and error logs. By combining these primitives, you can implement a relatively complete set of test cases.

Finally, I have a question for you to ponder: What are the advantages and disadvantages of this kind of abstraction, test::nginx? I welcome your feedback and discussion. Feel free to share this article and engage in conversations with others to exchange ideas and thoughts.