03 Unveiling the Sub Projects Hidden Behind

03 Unveiling the Sub-Projects Hidden Behind #

Hello, I’m Wen Ming.

Let’s first reveal the answer to the last reflection question from the previous section - how can we extract Lua code from nginx.conf while maintaining its readability and maintainability?

The operation is actually quite simple.

First, create a directory named “lua” in the geektime working directory specifically for storing the code:

$ mkdir lua
$ cat lua/hello.lua
ngx.say("hello, world")

Then, modify the configuration in nginx.conf by changing “content_by_lua_block” to “content_by_lua_file”:

pid logs/nginx.pid;
events {
	worker_connections 1024;
}

http {
	server {
		listen 8080;
		location / {
			content_by_lua_file lua/hello.lua;
		}
	}
}

Finally, restart the OpenResty service:

$ sudo kill -HUP `cat logs/nginx.pid`

You can use curl to verify if the expected result is returned. As for changes to the Lua code later on, you can directly modify the file “hello.lua” instead of nginx.conf.

In fact, there are a few interesting points in the example above:

  1. The relative path used in “content_by_lua_file lua/hello.lua” - how does OpenResty find this Lua file?

  2. Changes to the Lua code only take effect after restarting the OpenResty service, which is obviously inconvenient for debugging. Is there a way to make changes take effect immediately?

  3. How can we add the folder containing Lua code to the OpenResty lookup path?

I encourage you to think about these questions yourself, and you can find the answers in the official documentation. This is also why I have always emphasized the importance of documentation.

Next, let’s answer these questions together. Let’s start with the first question. If the original given path is a relative path, OpenResty will take the -p PATH option in the command line arguments when starting up as the prefix and concatenate it with the relative path to form an absolute path. This way, the Lua file can be found successfully.

Now let’s move on to the second question. The Lua code is loaded on the first request and cached by default. So every time you modify the Lua source file, you must reload OpenResty for the changes to take effect. In fact, you can disable lua_code_cache in nginx.conf to avoid reloading. You can try this yourself. However, it is important to note that this method is only temporary and should only be used for development and debugging. If it’s a production deployment, remember to enable the cache, otherwise it will greatly affect performance.

For the last question, OpenResty provides the lua_package_path directive to set the lookup path for Lua modules. In the example above, we can set the lua_package_path to $prefix/lua/?.lua;;, where:

  • $prefix is the -p PATH option in the startup parameters;
  • /lua/?.lua represents all files with the .lua extension in the lua directory;
  • the last two semicolons represent the built-in code search path.

Directory Structure After Installing OpenResty #

After understanding the first hello world program, let’s continue to explore the directory structure of OpenResty after it is installed and what files it contains.

First, let’s use the -V option to see which directory OpenResty is installed in. The following output has omitted many module compilation parameters, which we will fill in later:

$ openresty -V
nginx version: openresty/1.13.6.2
built by clang 10.0.0 (clang-1000.10.44.4)
built with OpenSSL 1.1.0h  27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/Cellar/openresty/1.13.6.2/nginx ...

I installed it using brew, so the directory is /usr/local/Cellar/openresty/1.13.6.2/nginx, which may be different from your local environment. It mainly includes the following subdirectories: bin, luajit, lualib, nginx, and pod. Understanding the meaning of these directories is important and can help us learn OpenResty better. Let’s take a look at each of them.

First, let’s look at the most important bin directory:

$ ll /usr/local/Cellar/openresty/1.13.6.2/bin
total 320
-r-xr-xr-x  1 ming  admin    19K  3 27 12:54 md2pod.pl
-r-xr-xr-x  1 ming  admin    15K  3 27 12:54 nginx-xml2pod
lrwxr-xr-x  1 ming  admin    19B  3 27 12:54 openresty -> ../nginx/sbin/nginx
-r-xr-xr-x  1 ming  admin    62K  3 27 12:54 opm
-r-xr-xr-x  1 ming  admin    29K  3 27 12:54 resty
-r-xr-xr-x  1 ming  admin    15K  3 27 12:54 restydoc
-r-xr-xr-x  1 ming  admin   8.3K  3 27 12:54 restydoc-index

Inside this directory, we have the OpenResty CLI resty mentioned in the previous section, as well as the core executable file openresty, which is actually a symbolic link to nginx. As for the other tools in the directory, there is no suspense, they are all Perl scripts, just like resty.

Among them, opm is the package management tool, which can be used to manage various third-party packages. There will be a dedicated section to talk about it later. restydoc is our “old friend” mentioned in the first section. It is a documentation viewer tool provided by OpenResty, which allows you to view the usage documentation for OpenResty and NGINX:

$ restydoc -s ngx.say
$ restydoc -s proxy_pass

The above code gives two examples, querying the API of OpenResty and the directive of NGINX. This tool, restydoc, is of great help to server engineers who focus on development.

After browsing the bin directory, let’s take a look at the pod directory. It should be noted that the “pod” here has nothing to do with the concept of “pod” in Kubernetes. Pod is a markup language in Perl, used to write documentation for Perl modules. The pod directory contains the documentation of OpenResty, NGINX, lua-resty-*, and LuaJIT, which are connected to what was mentioned about restydoc earlier.

Next, we have the familiar nginx and luajit directories. These two are self-explanatory, mainly containing the executable files and dependencies of NGINX and LuaJIT, which are the foundation of OpenResty. Many people say that OpenResty is based on Lua, but this is not entirely accurate. As we can see from above, OpenResty is actually based on LuaJIT.

In fact, in the early days, OpenResty included both Lua and LuaJIT, and you could decide whether to use Lua or LuaJIT through compilation options. However, Lua has gradually been phased out, and only LuaJIT, which has higher performance, is supported now.

Finally, let’s take a look at the lualib directory. It contains the Lua libraries used in OpenResty, mainly divided into two subdirectories: ngx and resty.

  • The former contains Lua code from the official project lua-resty-core. These are OpenResty APIs reimplemented based on FFI, which I will explain in a separate article later. You should have a rough impression here, and you don’t need to delve into it.

  • The resty directory contains Lua code from various lua-resty-* projects that we will encounter later.

As usual in my lectures, at this point, I will provide the source of these directories. This is also one of the joys of open source projects. If you like to dig deeper, you always find more interesting things.

The following is the packaging script of OpenResty in CentOS, which contains all the directories mentioned above. You can explore it yourself.

%files
%defattr(-,root,root,-)

/etc/init.d/%{name}
/usr/bin/%{name}
%{orprefix}/bin/openresty
%{orprefix}/site/lualib/
%{orprefix}/luajit/*
%{orprefix}/lualib/*
%{orprefix}/nginx/html/*
%{orprefix}/nginx/logs/
%{orprefix}/nginx/sbin/*
%{orprefix}/nginx/tapset/*
%config(noreplace) %{orprefix}/nginx/conf/*
%{orprefix}/COPYRIGHT

Overview of the OpenResty Project #

When it comes to OpenResty, you might think of the lua-nginx-module. Yes, this C module for NGINX is indeed the core of OpenResty, but it is not equivalent to OpenResty. Many engineers call OpenResty ngx lua, and this naming is used in many technical conference presentations and books. However, this is not rigorous and not advocated by the OpenResty community.

Let me explain why, and also talk about other associated projects in OpenResty besides lua-nginx-module.

Open the project homepage of OpenResty on GitHub, and you can see that OpenResty consists of 68 public projects, roughly divided into the following 7 categories. I will briefly introduce each of them, so that you can have a preliminary impression and find it easier to learn later.

NGINX C Modules #

OpenResty follows a naming convention for its projects, and those named with *-nginx-module are NGINX C modules.

OpenResty includes more than 20 C modules in total. You can also see these C modules in the openresty -V command we used at the beginning of this section:

$ openresty -V
nginx version: openresty/1.13.6.2
built by clang 10.0.0 (clang-1000.10.44.4)
built with OpenSSL 1.1.0h  27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/usr/local/Cellar/openresty/1.13.6.2/nginx --with-cc-opt='-O2 -I/usr/local/include -I/usr/local/opt/pcre/include -I/usr/local/opt/openresty-openssl/include' --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt='-Wl,-rpath,/usr/local/Cellar/openresty/1.13.6.2/luajit/lib -L/usr/local/lib -L/usr/local/opt/pcre/lib -L/usr/local/opt/openresty-openssl/lib' --pid-path=/usr/local/var/run/openresty.pid --lock-path=/usr/local/var/run/openresty.lock --conf-path=/usr/local/etc/openresty/nginx.conf --http-log-path=/usr/local/var/log/nginx/access.log --error-log-path=/usr/local/var/log/nginx/error.log --with-pcre-jit --with-ipv6 --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-dtrace-probes --with-stream --with-stream_ssl_module --with-http_ssl_module

The text after --add-module= is the OpenResty C module. Among them, the most important ones are lua-nginx-module and stream-lua-nginx-module. The former is used to handle layer 7 traffic, while the latter is used to handle layer 4 traffic.

Some of these C modules need special attention. Although they are compiled into OpenResty by default, they are not recommended to be used. For example, redis2-nginx-module, redis-nginx-module, and memc-nginx-module are used to interact with Redis and Memcached. These C libraries were recommended for use in the early stages of OpenResty, but after cosocket functionality was added, they were replaced by lua-resty-redis and lua-resty-memcached and are now in a state of limited maintenance.

OpenResty will not develop more NGINX C libraries in the future, but will focus on Lua libraries based on cosocket, which is the future direction.

lua-resty Peripheral Libraries #

The official repository of OpenResty contains 18 lua-resty-* libraries, covering common libraries for Redis, MySQL, Memcached, WebSocket, DNS, traffic control, string manipulation, in-process caching, etc. Besides the built-in official libraries, there are also many third-party libraries. These libraries are very important, so we will spend more time in the next chapter to introduce them specifically.

Custom Maintained LuaJIT Branch #

Besides maintaining their own OpenSSL patch, OpenResty also maintains their own LuaJIT branch. In 2015, the author of LuaJIT, Mike Pall, announced his retirement and searched for a new maintainer for LuaJIT. However, Mike did not find a suitable maintainer, so he currently mainly does bugfix maintenance and development of new features has been suspended. Therefore, OpenResty maintains its own branch of LuaJIT.

Compared to Lua, LuaJIT has added many unique functions which are very important, but only a few engineers are aware of them. They can be considered as a “semi-hidden skill”, and I will specifically introduce them later.

Testing Framework #

OpenResty’s testing framework is test-nginx, which is also developed in Perl. As the name suggests, it is specifically designed for testing NGINX-related projects. All C modules and lua-resty libraries from OpenResty’s official repositories are tested using test-nginx.

This framework is different from common assertion-based frameworks, as it is a more powerful and independent system. We will cover this topic in more detail in the following lessons.

In fact, some OpenResty contributors are not familiar with this testing framework. Sometimes, pull requests include complex C and Lua code, but they often neglect to write corresponding test cases. So, if you have looked at the test cases in the /t directory of some OpenResty projects and still feel confused, don’t doubt yourself just yet - most people feel the same way.

Apart from test-nginx, there is another project called mockeagain, which can simulate slow networks by allowing the program to read and write one byte at a time. This is a useful tool for web servers.

Debugging Toolchain #

OpenResty has devoted a lot of effort to scientific and dynamic code debugging, and it has reached an excellent level. The author of OpenResty, Yichun Zhang, wrote an article specifically to introduce dynamic tracing techniques. I highly recommend reading it, as it will help you understand the corresponding toolchain.

Two OpenResty projects, openresty-systemtap-toolkit and stapxx, are based on systemtap, a dynamic debugging and tracing tool. The main advantage of using systemtap is that it enables live analysis without any interference with the target program.

To put it simply, systemtap is like getting a painless and non-intrusive CT scan at a hospital. What’s even better is that systemtap can generate intuitive flame graphs for performance analysis. I will explain this in more detail later, but for now, here is a flame graph to give you a visual idea:

The packaging scripts for OpenResty on different operating systems (such as CentOS, Ubuntu, MacOS, etc.) are manually written to achieve greater control. We have already touched on these packaging-related projects when introducing the directory structure after installation: openresty-packaging and home-brew. If you are interested, you can study them further, but I won’t elaborate on them here.

Engineering Tools #

In addition to the major projects mentioned above, OpenResty also has several tools responsible for engineering, most of which are “hidden treasures.”

For example, openresty-devel-utils is a toolkit for developing OpenResty and NGINX. These tools are also developed in Perl, and most of them have no documentation. However, they are very useful for OpenResty developers.

Let me briefly introduce a few of them.

lj-releng is a simple and effective LuaJIT code inspection tool, similar to luacheck. It can identify potential issues such as global variables.

reindex is a tool that, as the name suggests, can rebuild indexes. In reality, it is used to format test-nginx test cases, reordering their numbers and removing unnecessary whitespace. Reindex is one of the tools that OpenResty developers use almost every day.

opsboy is another hidden project primarily used for automated deployments. Before each OpenResty release, a comprehensive regression test is performed on an AWS EC2 cluster. You can refer to the official documentation for detailed information. This regression test is deployed and driven by opsboy.

Opsboy is a Perl-based Domain-Specific Language (DSL). In fact, the author of OpenResty enjoys creating various DSLs to solve different problems.

In conclusion #

Today, we mainly studied the directory structure and some sub-projects behind OpenResty after installation. I hope that after learning today’s content, you can have a better understanding of the OpenResty project. OpenResty has gone far beyond the scope of NGINX load balancing and reverse proxy, and has built its own ecosystem. Next time, we will discuss this in more detail.

Do you have any doubts or questions about today’s content? Feel free to leave a message and share with me. You are also welcome to forward this article to your colleagues and friends to learn efficient development together.