26 Introduction to 'Testnginx' a Roadblock for Code Contributors

26 Introduction to ’testnginx’ - A Roadblock for Code Contributors #

Hello, I am Wen Ming.

Testing is an essential part of software development. The concept of Test-Driven Development (TDD) has become well-established, and almost every software company has a QA team responsible for testing.

Testing is also the cornerstone of OpenResty’s quality and good reputation. However, at the same time, it is also the most overlooked part of many open-source projects in OpenResty. Many developers use lua-nginx-module daily and occasionally run flame graphs, but how many of them actually run test cases? Even many open-source projects based on OpenResty do not have test cases. However, open-source projects without test cases and continuous integration are obviously not trustworthy.

However, unlike commercial companies, most open-source projects do not have dedicated testing engineers. So how do they ensure code quality? The answer is simple: “automated testing” and “continuous integration.” The key is automation and continuity, and OpenResty has achieved excellence in both aspects.

OpenResty has 70 open-source projects, and the workload of unit testing, integration testing, performance testing, mock testing, fuzz testing, and so on, cannot be handled solely by the community. Therefore, OpenResty invested heavily in automated testing from the beginning. This may slow down the project progress in the short term, but it can be said to be a one-time effort, and in the long run, the investment in this area is very cost-effective. Therefore, whenever I discuss OpenResty’s testing ideas and toolset with other engineers, they are always amazed.

Next, let’s talk about OpenResty’s testing philosophy.

Philosophy #

test::nginx is the core of the OpenResty test system. Both OpenResty itself and the surrounding lua-resty libraries use it to organize and write test suites. Although it is a testing framework, its entry barrier is very high. This is because test::nginx is different from general testing frameworks; it is not based on assertions and does not use the Lua language. This requires developers to learn and use test::nginx from scratch, and to overturn their inherent understanding of testing frameworks.

I know several contributors to OpenResty who can fluently submit C and Lua code, but they get stuck when writing test cases using test::nginx. Either they don’t know how to write them, or they don’t know how to solve the problem when the tests fail. Therefore, I call test::nginx a stumbling block for code contributors.

test::nginx combines Perl, data-driven testing, and DSL (domain-specific language). For the same set of test cases, different effects can be achieved by controlling the parameters and environment variables, such as random execution, repeated execution, memory leak detection, and stress testing.

Installation and Examples #

After discussing so many concepts, let’s have an intuitive understanding of test::nginx. Before using it, let’s first look at how to install it.

When it comes to installing software in the OpenResty ecosystem, the installation method in the official CI is the most timely and effective. Other installation methods always encounter various problems. So, I always recommend that you refer to its method in Travis.

The installation and usage of test::nginx are no exception. In Travis, it can be divided into four steps.

1. First, install Perl’s package manager cpanminus. - 2. Then, use cpanm to install test::nginx:

sudo cpanm --notest Test::Nginx IPC::Run > build.log 2>&1 || (cat build.log && exit 1)

3. Next, clone the latest source code:

git clone https://github.com/openresty/test-nginx.git

4. Finally, use Perl’s prove command to load the test-nginx library and run the test cases in the /t directory:

prove -Itest-nginx/lib -r t

After installation, let’s take a look at the simplest test case in test::nginx. The following code is adapted from the official documentation. I have removed all the personalized control parameters:

use Test::Nginx::Socket 'no_plan';


run_tests();

__DATA__

=== TEST 1: set Server
--- config
    location /foo {
        echo hi;
        more_set_headers 'Server: Foo';
    }
--- request
    GET /foo
--- response_headers
    Server: Foo
--- response_body
hi

Although test::nginx is written in Perl and is one of its modules, can you see any traces of Perl or any other language from the above test? If you have this feeling, then that’s right. Because test::nginx itself is a DSL (Domain Specific Language) implemented by the author using Perl, specifically designed for testing Nginx and OpenResty.

So, when you first see this kind of test, you probably won’t understand it. But don’t worry, let me explain the above test case for you.

Firstly, use Test::Nginx::Socket;, this is the way to import libraries in Perl, just like requiring modules in Lua. This also reminds us that test::nginx is a Perl program.

The second line run_tests(); is a Perl function in test::nginx, which is the entry function of the test framework. If you want to call other Perl functions in test::nginx, they must be placed before run_tests() to be effective.

The third line __DATA__ is a marker indicating that what follows is the test data. Perl functions should be defined before this marker.

Next, === TEST 1: set Server is the title of the test case, which indicates the purpose of this test. The numeric identifier inside it can be automatically sorted by tools.

--- config is the Nginx configuration segment. In the above example, we only used Nginx directives, without involving Lua. If you want to add Lua code, you can also use directives like content_by_lua here.

--- request is used to simulate sending a request from the terminal. The following GET /foo specifies the request method and URI.

--- response_headers is used to check the response headers. The following Server: Foo indicates the header and value that must appear in the response headers. If it does not appear, the test will fail.

Finally, --- response_body is used to check the response body. The following hi is the string that must appear in the response body. If it does not appear, the test will fail.

Alright, with this, we have analyzed the simplest test case. Do you understand it now? If there is anything unclear, be sure to ask questions in time. After all, being able to understand test cases is a prerequisite for completing OpenResty-related development work.

Write your own test case #

Practice what you preach. Now, it’s time to get hands-on and start experimenting. Do you remember how we tested the memcached server in the previous lesson? That’s right, we used resty to manually send requests, which can be represented by the following code:

$ resty -e 'local memcached = require "resty.memcached"
    local memc, err = memcached:new()

    memc:set_timeout(1000) -- 1 sec
    local ok, err = memc:connect("127.0.0.1", 11212)
    local ok, err = memc:set("dog", 32)
    if not ok then
        ngx.say("failed to set dog: ", err)
        return
    end

    local res, flags, err = memc:get("dog")
    ngx.say("dog: ", res)'

However, do you feel that manual testing is not smart enough? Don’t worry, after learning test::nginx, we can try to automate manual testing, like the following code:

use Test::Nginx::Socket::Lua::Stream;

run_tests();

__DATA__

=== TEST 1: basic get and set
--- config
        location /test {
            content_by_lua_block {
                local memcached = require "resty.memcached"
                local memc, err = memcached:new()
                if not memc then
                    ngx.say("failed to instantiate memc: ", err)
                    return
                end

                memc:set_timeout(1000) -- 1 sec
                local ok, err = memc:connect("127.0.0.1", 11212)

                local ok, err = memc:set("dog", 32)
                if not ok then
                    ngx.say("failed to set dog: ", err)
                    return
                end

                local res, flags, err = memc:get("dog")
                ngx.say("dog: ", res)
            }
        }

--- stream_config
    lua_shared_dict memcached 100m;

--- stream_server_config
    listen 11212;
    content_by_lua_block {
        local m = require("memcached-server")
        m.go()
    }

--- request
GET /test
--- response_body
dog: 32
--- no_error_log
[error]

In this test case, I added --- stream_config, --- stream_server_config, and --- no_error_log configuration items, but essentially they are all the same, which is:

By abstracting well-defined primitives (also considered as configurations), we can separate the test data and the detection process, making it more readable and extensible.

This is the fundamental difference between test::nginx and other testing frameworks. This DSL is a double-edged sword. It can make the test logic clear and easy to extend, but at the same time, it raises the learning curve as you need to learn new syntax and configurations to start writing test cases.

Conclusion #

I have to say that test::nginx is powerful, but often it may not be suitable for your scenario. Why use a sledgehammer to crack a nut? In OpenResty, you can also choose to use the busted testing framework with an assertion style. busted, combined with the command-line tool resty, can also meet many testing needs.

Finally, I’ll leave you with a homework question. Can you run this memcached test locally? It would be even better if you can add a new test case.

Feel free to record your actions and experiences in the comments section, and you can also write down any doubts you have while studying today. Also, feel free to share this article with more people who are interested in OpenResty. Let’s communicate and discuss together.