18 the Most Important Data Structure for Communication Between Workers Shared Dict

18 The Most Important Data Structure for Communication Between Workers- shared dict #

Hello, I’m Wen Ming.

As we mentioned before, in Lua, table is the only data structure. Correspondingly, the shared memory dictionary, shared dict, is the most important data structure in OpenResty programming. It not only supports the storage and retrieval of data but also supports atomic counting and queue operations.

With shared dict, you can implement caching and communication between multiple workers, as well as functions such as rate limiting, traffic statistics, and more. You can use shared dict as a simplified version of Redis. The only difference is that the data stored in shared dict cannot be persisted, so you must consider the possibility of data loss.

Several Ways of Data Sharing #

During the process of writing OpenResty Lua code, you will inevitably encounter situations where data needs to be shared between different stages of a request or between different workers, and you may also need to share data between Lua and C code.

So, before formally introducing the API of shared dict, let’s first understand several common methods of data sharing in OpenResty, and learn how to choose the most suitable method based on the actual situation.

The first method is using variables in Nginx. It can share data between Nginx C modules, and naturally, it can also share data between C modules and the lua-nginx-module provided by OpenResty, like the following code:

location /foo {
     set $my_var ''; # this line is required to create $my_var at config time
     content_by_lua_block {
         ngx.var.my_var = 123;
         ...
     }
 }

However, using Nginx variables to share data is relatively slow because it involves hash lookup and memory allocation. At the same time, this method has its limitations and can only be used to store strings, and cannot support complex Lua types.

The second method is ngx.ctx, which can share data between different stages of the same request. It is actually a normal Lua table, so it is very fast and can also store various Lua objects. Its lifespan is request-level, which means that when a request ends, ngx.ctx will also be destroyed along with it.

Here is a typical use case where we use ngx.ctx to cache expensive calls like Nginx variables, and it can be used in different stages:

location /test {
     rewrite_by_lua_block {
         ngx.ctx.host = ngx.var.host
     }
     access_by_lua_block {
        if (ngx.ctx.host == 'openresty.org') then
            ngx.ctx.host = 'test.com'
        end
     }
     content_by_lua_block {
         ngx.say(ngx.ctx.host)
     }
 }

In this case, if you access it using curl:

curl -i 127.0.0.1:8080/test -H 'host:openresty.org'

It will print out test.com, indicating that ngx.ctx does indeed share data between different stages. Of course, you can also modify the example above yourself to store more complex objects such as tables, instead of simple strings, to see if it meets your expectations.

However, it needs to be noted that since the lifespan of ngx.ctx is request-level, it cannot be used for caching at the module level. For example, using it like this in the foo.lua file is incorrect:

local ngx_ctx = ngx.ctx

local function bar()
    ngx_ctx.host =  'test.com'
end

We should call and cache it at the function level:

local ngx = ngx

local function bar()
    ngx_ctx.host =  'test.com'
end

There are many more details about ngx.ctx, which we will continue to explore in the performance optimization section later.

Continuing down, the third method is using module-level variables to share data between all requests within the same worker. Unlike Nginx variables and ngx.ctx mentioned earlier, this method is a bit harder to understand. But don’t worry, let’s first take a look at an example to understand what module-level variables are:

-- mydata.lua
 local _M = {}

 local data = {
     dog = 3,
     cat = 4,

 local _M = {}
 
 local data = {
     dog = 3,
     cat = 4,
     pig = 5,
 }
 
 function _M.get_age(name)
     return data[name]
 end
 
 return _M

In the nginx.conf configuration:

location /lua {
     content_by_lua_block {
         local mydata = require "mydata"
         ngx.say(mydata.get_age("dog"))
     }
 }

In this example, mydata is a module that is only loaded once by the worker process. After that, all requests handled by this worker will share the code and data of the mydata module.

Naturally, the data variable in the mydata module is a module-level variable, located at the top level of the module, accessible by all functions.

Therefore, you can place data that needs to be shared between requests in module-level variables. However, it is important to note that we usually use this approach to store read-only data. If there are write operations involved, you need to be very careful because there may be race conditions, which are very difficult to debug bugs.

Let’s understand this with the simplest example possible:

-- mydata.lua
 local _M = {}

 local data = {
     dog = 3,
     cat = 4,
     pig = 5,
 }

 function _M.incr_age(name)
     data[name] = data[name] + 1
    return data[name]
 end

 return _M

In the module, we added the incr_age function, which modifies the data in the data table.

Then, in the calling code, we added the crucial line ngx.sleep(5), which is a yield operation:

location /lua {
     content_by_lua_block {
         local mydata = require "mydata"
         ngx.say(mydata.incr_age("dog"))
         ngx.sleep(5) -- yield API
         ngx.say(mydata.incr_age("dog"))
     }
 }

If there is no sleep code (or any other non-blocking IO operation, such as accessing Redis), there will be no yield operation and no competition, so the final output number will be sequential.

However, when we add this line of code, even within the 5 seconds of sleep, it is highly likely that another request will call the mydata.incr_age function and modify the value of the variable, resulting in non-sequential output. Keep in mind that in actual code, the logic will not be this simple, making bug debugging much more difficult.

Therefore, unless you are certain that there is no yield operation in between, meaning control is not handed over to the Nginx event loop, I recommend keeping module-level variables as read-only.

The fourth and final method is using a shared dict to share data, and this data can be shared between multiple workers.

This method is based on a red-black tree implementation and has good performance, but it also has its limitations. You must declare the size of shared memory in Nginx’s configuration file in advance, and this cannot be changed during runtime:

lua_shared_dict dogs 10m;

Shared dict can only cache string-type data and does not support complex Lua data types. This means that when I need to store complex data types such as tables, I will have to use JSON or other methods to serialize and deserialize, which naturally incurs some performance overhead.

In conclusion, there is no one-size-fits-all data sharing method. You need to combine multiple methods based on your requirements and scenarios.

Shared Dictionary #

In the previous section on data sharing, we spent a lot of time learning about it. Some people may wonder: they seem unrelated to shared dicts, are they off-topic?

The truth is that they are not. You can think about why OpenResty needs the existence of shared dicts.

Let’s recall the various methods we discussed earlier. The scope of data sharing for the first three methods is at the request level or the level of a single worker. Therefore, in the current implementation of OpenResty, only the shared dict can achieve data sharing between workers and facilitate communication between workers. This is the value of its existence.

In my opinion, it is more important to understand why a technology exists and clarify its differences and advantages compared to similar technologies than just being proficient in calling its provided APIs. This kind of technological perspective will bring you a certain degree of foresight and insight, which can also be regarded as an important distinction between engineers and architects.

Returning to the shared dictionary itself, it provides more than 20 Lua APIs. However, all these APIs are atomic operations, so you don’t have to worry about competition issues between multiple workers and high concurrency.

All these APIs have detailed official documentation, so I won’t go into detail here. I would like to emphasize that no matter how much you learn from a technology course, it can’t replace careful reading of the official documentation. This time-consuming effort cannot be skipped by anyone.

Let’s continue looking at the APIs of the shared dict. These APIs can be divided into the following three categories: dictionary read/write, queue operations, and management.

Dictionary Read/Write #

First, let’s look at the dictionary read/write category. In the initial version, only dictionary read/write APIs were available, and they are also the most commonly used functions of shared dict. Here is a simple example:

$ resty --shdict='dogs 1m' -e 'local dict = ngx.shared.dogs
                               dict:set("Tom", 56)
                               print(dict:get("Tom"))'

In addition to set, OpenResty also provides safe_set, add, and safe_add methods for writing data. The prefix safe means that in the case of memory being full, it won’t evict old data based on LRU but instead returns an error message of “no memory” if the write fails.

In addition to get, OpenResty also provides a method get_stale to read data. Compared to the get method, it returns an additional value indicating whether the data has expired:

value, flags, stale = ngx.shared.DICT:get_stale(key)

You can also call the delete method to remove a specific key, which is equivalent to set(key, nil).

Queue Operations #

Let’s look at queue operations, which are additional features added to OpenResty later and provide an interface similar to Redis. Each element in the queue is described using ngx_http_lua_shdict_list_node_t:

typedef struct { 
    ngx_queue_t queue; 
    uint32_t value_len; 
    uint8_t value_type; 
    u_char data[1]; 
} ngx_http_lua_shdict_list_node_t;

I have included the PR for these queue operation APIs in the article. If you are interested, you can analyze the specific implementation by following the documentation, test cases, and source code.

However, the following five queue APIs do not have corresponding code examples in the documentation, so I will briefly introduce them here:

  • lpush/rpush: Add elements at both ends of the queue.
  • lpop/rpop: Pop elements from both ends of the queue.
  • llen: Return the number of elements in the queue.

Don’t forget the powerful tool we discussed in the previous lesson - test cases. If there are no examples in the documentation, we can usually find the corresponding code in the test cases. The tests related to queues are in the file 145-shdict-list.t:

=== TEST 1: lpush & lpop
--- http_config
    lua_shared_dict dogs 1m;
--- config
    location = /test {
        content_by_lua_block {
            local dogs = ngx.shared.dogs

            local len, err = dogs:lpush("foo", "bar")
            if len then
                ngx.say("push success")
            else
                ngx.say("push err: ", err)
            end

            local val, err = dogs:llen("foo")
            ngx.say(val, " ", err)

            local val, err = dogs:lpop("foo")
            ngx.say(val, " ", err)

            local val, err = dogs:llen("foo")
            ngx.say(val, " ", err)

            local val, err = dogs:lpop("foo")
            ngx.say(val, " ", err)
        }
    }
--- request
GET /test
--- response_body
push success
1 nil
bar nil
0 nil
nil nil
--- no_error_log
[error]

Management #

Finally, let’s talk about the management APIs, which were added later and are in high demand in the community. Among them, the usage of shared memory is the most typical example. For example, if a user requests 100M of space as a shared dict, is this 100M sufficient? How many keys are stored in it? What are the specific keys? These are all very practical questions.

Regarding these types of questions, the official stance of OpenResty is to encourage users to use flame graphs to solve them, which is non-intrusive and keeps the code base efficient and clean, rather than providing intrusive APIs to directly return results.

However, from the perspective of user-friendliness, these management APIs are still very necessary. After all, open-source projects are used to solve product requirements, not to showcase the technology itself. So, let’s take a look at these management APIs that were added later.

First is get_keys(max_count?), which by default returns only the first 1024 keys. If you set max_count to 0, it will return all keys.

Then there are capacity and free_space, which are both part of the lua-resty-core repository, so you need to require them before using:

require "resty.core.shdict"

 local cats = ngx.shared.cats
 local capacity_bytes = cats:capacity()
 local free_page_bytes = cats:free_space()

They respectively return the size of the shared memory (the size configured in lua_shared_dict) and the number of free bytes in the pages. Because shared dicts allocate space by pages, even if free_space returns 0, there may still be space available within the allocated pages, so its return value cannot represent the actual occupancy of shared memory.

Closing Remarks #

In practical development, we often make use of multi-level caching. OpenResty’s official projects also have encapsulations for caching. Can you find out which projects they are? Or do you know any other lua-resty libraries that provide caching encapsulation?

Feel free to leave a comment and share your thoughts with me. You are also welcome to share this article with your colleagues and friends. Let’s communicate and improve together.