31 The Truly Malicious Function that Causes a 10x Performance Drop #

Hello, I’m Wen Ming.

Through the study of the previous chapters, I believe you already have a comprehensive understanding of the architecture of LuaJIT and OpenResty, as well as Lua API and testing. Now, we are about to enter the chapter with the most content and the easiest to overlook: performance optimization.

In the performance optimization chapter, I will familiarize you with all aspects of performance optimization in OpenResty, and summarize the scattered content mentioned in the previous chapters into a comprehensive coding guide for OpenResty, so that you can write higher quality OpenResty code.

You should know that improving performance is not easy. You need to consider system architecture optimization, database optimization, code optimization, performance testing, flame graph analysis, and many other steps. On the contrary, decreasing performance is easy. Just like the title of today’s lesson, you only need to add a few lines of code to make the performance drop by 10 times or even more. If you use OpenResty to write code but the performance never improves, then it’s very likely that you are using blocking functions.

Therefore, before introducing specific methods of performance optimization, let’s first understand an important principle in OpenResty programming: avoid using blocking functions.

Since childhood, we have been taught by our parents and teachers not to play with fire, not to touch electrical plugs, as these are dangerous behaviors. Likewise, there are such dangerous behaviors in OpenResty programming. If your code involves blocking operations, it will lead to a sharp decline in performance, thus undermining the original intention of using OpenResty to build high-performance servers.

Why not use blocking operations? #

Understanding which behaviors are dangerous and avoiding using them is the first step in performance optimization. Let’s first review why blocking operations affect the performance of OpenResty.

The reason why OpenResty can maintain high performance is simply because it borrows the event handling of Nginx and the coroutine mechanism of Lua. So:

  - When encountering operations such as network I/O that need to wait for a return before continuing, it will first call the yield function of the Lua coroutine to suspend itself, and then register a callback in Nginx.   - After the I/O operation is completed (it may also be a timeout or an error), Nginx callback will resume to wake up the Lua coroutine.

This process ensures that OpenResty can efficiently use CPU resources to handle all requests.

In this processing flow, if blocking functions are used instead of non-blocking methods like cosocket to handle I/O, LuaJIT will not transfer control to the event loop of Nginx. This will result in other requests waiting in line for the blocking event to be processed before receiving a response.

In summary, in the programming of OpenResty, we should be particularly cautious with function calls that may cause blocking; otherwise, a single line of blocking code can bring down the performance of the entire service.

Next, let me introduce a few common pitfalls, which are frequently misused blocking functions. Let’s also understand how to “destroy” your service performance quickly in the simplest way, causing its performance to drop by 10 times.

Executing External Commands #

In many scenarios, developers do not only use OpenResty as a web server, but also incorporate additional business logic into it. In such cases, it may be necessary to invoke external commands and tools to assist in performing certain operations.

For example, killing a process:

os.execute("kill -HUP " .. pid)

Or copying files, generating keys with OpenSSL, or performing other time-consuming operations:

os.execute(" cp test.exe /tmp ")

os.execute(" openssl genrsa -des3 -out private.pem 2048 ")

On the surface, os.execute is a built-in function in Lua, and in the Lua world, it is indeed used to call external commands in this way. However, we must remember that Lua is an embedded language and has completely different recommended usage in different contexts.

In the OpenResty environment, os.execute blocks the current request. Therefore, if the execution time of this command is very short, the impact is not significant. However, if the command takes several hundred milliseconds or even seconds to execute, the performance will deteriorate significantly.

Now that we understand the problem, how can we solve it? Generally, there are two solutions.

Solution 1: If an FFI library is available, use it preferentially to make the call. #

For example, instead of using OpenSSL’s command line to generate keys, you can use FFI to directly call OpenSSL’s C functions to bypass it.

As for the example of killing a process, you can use the lua-resty-signal library provided by OpenResty to accomplish it non-blockingly. The code implementation is as follows. Of course, lua-resty-signal actually uses FFI to call system functions to solve the problem.

local resty_signal = require "resty.signal"
local pid = 12345

local ok, err = resty_signal.kill(pid, "KILL")

Additionally, on the LuaJIT official website, there is a dedicated page that categorically introduces various FFI binding libraries. When dealing with CPU-intensive operations such as image processing or encryption/decryption, you can refer to this page to see if there are already encapsulated libraries that can be directly used.

Solution 2: Use the lua-resty-shell library based on ngx.pipe. #

As mentioned before, you can run your own command in shell.run, which is a non-blocking operation:

$ resty -e 'local shell = require "resty.shell"
local ok, stdout, stderr, reason, status =
    shell.run([[echo "hello, world"]])
    ngx.say(stdout)'

Disk I/O #

Now let’s take a look at scenarios involving disk I/O. Reading a local configuration file is a common operation in a server-side program, like in the code snippet below:

local path = "/conf/apisix.conf"
local file = io.open(path, "rb")
local content = file:read("*a")
file:close()

This code uses io.open to retrieve the contents of a file. Although it is a blocking operation, it is important to consider the actual scenario. If you call this code in the init or init worker phase, it is actually a one-time action and does not affect any end-user requests, making it acceptable.

However, if each user request triggers disk read/write operations, it becomes unacceptable. In that case, you need to seriously consider a solution.

The first approach is using the third-party C module lua-io-nginx-module. It provides a “non-blocking” Lua API for OpenResty. However, this “non-blocking” is enclosed in quotation marks because you cannot use it as freely as cosockets. Disk I/O consumption does not disappear magically; it just changes the way it is performed.

The principle behind this approach is that lua-io-nginx-module utilizes Nginx’s thread pool to move disk I/O operations to another thread, allowing the main thread to remain unblocked.

When using this library, you need to recompile Nginx as it is a C module. Its usage is as follows, which is similar to Lua I/O library:

local ngx_io = require "ngx.io"
local path = "/conf/apisix.conf"
local file, err = ngx_io.open(path, "rb")
local data, err = file:read("*a")
file:close()

The second approach is to consider architectural adjustments. Can we find another way to avoid reading/writing to the local disk for this type of disk I/O?

Here’s an example for you to consider. Several years ago, in a project I worked on, we needed to log information to the local disk for statistical and troubleshooting purposes.

At that time, the developers used ngx.log to write these logs, like this:

ngx.log(ngx.WARN, "info")

This line of code uses the Lua API provided by OpenResty, and it seems fine. However, the downside is that you cannot call it frequently. First, ngx.log itself is an expensive function call. Second, even with a buffer, a large number of frequent disk writes significantly impact performance.

So, how can we solve this issue? Let’s go back to the original requirements: statistics and troubleshooting. Writing to the local disk is just one means of achieving this goal.

Therefore, you can also send the logs to a remote log server, allowing you to use cosockets for non-blocking network communication and not block the external services with blocking disk I/O. You can use lua-resty-logger-socket to accomplish this:

local logger = require "resty.logger.socket"
if not logger.initted() then
    local ok, err = logger.init{
        host = 'xxx',
        port = 1234,
        flush_limit = 1234,
        drop_limit = 5678,
    }
local msg = "foo"
local bytes, err = logger.log(msg)

In fact, you may have noticed that the essence of the two methods mentioned above is the same: if blocking cannot be avoided, do not block the main working thread; instead, hand it off to external threads or services.

luasocket #

Finally, let’s talk about luasocket, which is also a built-in Lua library that developers often use. Sometimes people confuse luasocket with the cosocket provided by OpenResty. While luasocket can also be used for network communication, it does not have the advantages of non-blocking operations. If you use luasocket, performance will decrease significantly.

However, luasocket still has its unique use cases. Do you remember what we mentioned before? In some stages where cosocket cannot be used, we can generally use the ngx.timer method to work around it. Also, you can use luasocket to achieve the functionality of cosocket in one-time stages such as init_by_lua* and init_worker_by_lua*. The more familiar you are with the similarities and differences between OpenResty and Lua, the more interesting solutions like this you will be able to find.

In addition, lua-resty-socket is actually a second-level encapsulation of an open-source library that achieves compatibility between luasocket and cosocket. This content is worth further study. If you have the ability to do so, I have prepared some resources for you to continue learning here.

Conclusion #

In conclusion, in OpenResty, understanding the types and solutions to blocking operations is the foundation for effective performance optimization. So, have you encountered similar blocking operations in your actual development? How did you discover and solve them? Feel free to leave a comment and share your experiences with me. Also, please feel free to share this article with others.