04 How to Manage Third Party Packages With Package Management Tools Luarocks and Opm

04 How to Manage Third-party Packages with Package Management Tools luarocks and opm #

Hello, I am Wen Ming.

In the previous section, we briefly learned about some of the official projects of OpenResty. However, if we use OpenResty in a production environment, it is obvious that the libraries that come with the OpenResty installation package are not enough. For example, there is no lua-resty library for making HTTP requests, and there is no way to interact with Kafka.

So what should we do? In this section, let’s explore together where we can find these third-party libraries.

Here, I want to emphasize once again that OpenResty is not a fork of NGINX, nor is it a repackage of NGINX with some commonly used libraries added on top. It simply uses NGINX as a underlying network library.

When you use NGINX, you don’t think about how to make custom HTTP requests or how to interact with Kafka. In the world of OpenResty, thanks to the existence of cosocket, developers can easily write libraries like lua-resty-http and lua-resty-kafka to handle such requirements, just like using development languages like Python or PHP.

Additionally, I have another suggestion for you: you should not use any Lua world libraries to solve the aforementioned problems, but instead you should use lua-resty-* libraries that work with cosocket. Lua world libraries are likely to bring blocking operations, which directly reduce the performance of the service by several orders of magnitude. This is a common mistake made by beginners in OpenResty, and it is not easily noticeable.

So how do we find these non-blocking lua-resty-* libraries? Next, let me introduce several ways for you.

OPM #

OPM (OpenResty Package Manager) is the package manager that comes with OpenResty and can be used directly once you have installed OpenResty. We can try searching for a library that sends HTTP requests using the command $ opm search http.

The first query may be slow and take a few seconds. opm.openresty.org will perform a query from the PostgreSQL database and cache the results for a period of time. The specific search results are quite long, but here we will only look at the first result:

openresty/lua-resty-upload    			Streaming reader and parser for HTTP file uploading based on ngx_lua cosocket

Hmm, looking at this result, you may wonder: what does the lua-resty-upload package have to do with sending HTTP requests?

It turns out that when OPM performs a search, it searches both the package name and the package summary using the keywords provided. This is also why the search above takes a few seconds, because it performs a full-text search on the strings in PostgreSQL.

However, regardless of that, the returned result is not very user-friendly. Let’s modify the keywords and search again:

$ opm search lua-resty-http
ledgetech/lua-resty-http                          Lua HTTP client cosocket driver for OpenResty/ngx_lua
pintsized/lua-resty-http                          Lua HTTP client cosocket driver for OpenResty/ngx_lua
agentzh/lua-resty-http                            Lua HTTP client cosocket driver for OpenResty/ngx_lua

In the world of OpenResty, if you implement a package using cosocket, you should use the prefix lua-resty-. It’s an unwritten rule.

Looking back at the previous search results, OPM uses the contributor’s GitHub repository address as the package name, which is the GitHub ID / repo name. The search above returned three third-party libraries for lua-resty-http. Which one should we choose?

If you have sharp eyes, you may have noticed the ID agentzh, and you would be correct. That is OpenResty’s creator, the great Spring Hackerman himself. Before choosing this package, let’s take a look at the number of stars and the last update time: it only has a few stars, and the last update was in 2016. Clearly, this is an abandoned project. Taking a closer look, pintsized/lua-resty-http and ledgetech/lua-resty-http actually point to the same repository. So, it doesn’t matter which one you choose.

Additionally, the OPM website is relatively simple, and it does not provide download counts or package dependencies. You will need to spend more time to determine the correct choice of lua-resty libraries, which should be the responsibility of the maintainers.

LUAROCKS #

LuaRocks is another package manager in the OpenResty world, created before OPM. Unlike OPM, which only includes packages related to OpenResty, LuaRocks also includes libraries from the Lua world. For example, LuaSQL-MySQL in LuaRocks is a Lua package for connecting to MySQL, but it cannot be used in OpenResty.

Let’s use LuaRocks to search for an HTTP library as an example:

$ luarocks search http

You can see that it returns a lot of packages.

Let’s try using a different keyword:

$ luarocks search lua-resty-http

This time, it only returns one package. We can go to the LuaRocks website to check out the detailed information for this package. Here is a screenshot of the website page:

It includes the author, license, GitHub address, download count, brief description, version history, and dependencies. Unlike OPM, LuaRocks does not directly use GitHub user information, but requires developers to register separately on LuaRocks.

In fact, Kong, an open-source API gateway project, uses LuaRocks for package management and even brings LuaRocks’ author into their team. Let’s take a brief look at how Kong’s package management configuration is written.

The latest version of Kong is 1.1.1, and you can find the latest .rockspec file at https://github.com/Kong/kong.

package = "kong"
version = "1.1.1-0"
supported_platforms = {"linux", "macosx"}
source = {
  url = "git://github.com/Kong/kong",
  tag = "1.1.1"
}
description = {
  summary = "Kong is a scalable and customizable API Management Layer built on top of Nginx.",
  homepage = "https://konghq.com",
  license = "Apache 2.0"
}
dependencies = {
  "inspect == 3.1.1",
  "luasec == 0.7",
  "luasocket == 3.0-rc1",
  "penlight == 1.5.4",
  "lua-resty-http == 0.13",
  "lua-resty-jit-uuid == 0.0.7",
  "multipart == 0.5.5",
  "version == 1.0.1",
  "kong-lapis == 1.6.0.1",
  "lua-cassandra == 1.3.4",
  "pgmoon == 1.9.0",
  "luatz == 0.3",
  "http == 0.3",
  "lua_system_constants == 0.1.3",
  "lyaml == 6.2.3",
  "lua-resty-iputils == 0.3.0",
  "luaossl == 20181207",
  "luasyslog == 1.0.0",
  "lua_pack == 1.0.5",
  "lua-resty-dns-client == 3.0.2",
  "lua-resty-worker-events == 0.3.3",
  "lua-resty-mediador == 0.1.2",
  "lua-resty-healthcheck == 0.6.0",
  "lua-resty-cookie == 0.1.0",
  "lua-resty-mlcache == 2.3.0",
  ......

From the file, you can see that the dependencies include both lua-resty libraries and libraries from the Lua world. Only some of these dependencies can be installed using OPM. After configuring the file, developers can use the luarocks upload command to upload the configuration, allowing users to download and install Kong using LuaRocks.

In addition, in OpenResty, besides Lua code, we often need to call C code. This requires compilation to use. LuaRocks supports this. You can specify the path and name of the C source code in the rockspec file, and LuaRocks will help you compile it locally. However, OPM does not currently support this feature.

However, it is important to note that both OPM and LuaRocks do not support private packages.

AWESOME-RESTY #

After discussing package management in OpenResty, we still have limited access to the lua-resty packages, even with OPM and LuaRocks. Is there a place where we can see a complete list of available packages?

The answer is yes, the awesome-resty project maintains a comprehensive list of almost all the packages available for OpenResty, neatly categorized. When you are unsure if a suitable third-party package exists, this is the best place to look.

Let’s use the HTTP library as an example. In awesome-resty, it belongs to the networking category:

lua-resty-http by @pintsized — Lua HTTP client cosocket driver for OpenResty / ngx_lua
lua-resty-http by @liseen — Lua http client driver for the ngx_lua based on the cosocket API
lua-resty-http by @DorianGray — Lua HTTP client driver for ngx_lua based on the cosocket API
lua-resty-http-simple — Simple Lua HTTP client driver for ngx_lua
lua-resty-httpipe — Lua HTTP client cosocket driver for OpenResty / ngx_lua
lua-resty-httpclient — Nonblocking Lua HTTP Client library for aLiLua & ngx_lua
lua-httpcli-resty — Lua HTTP client module for OpenResty
lua-resty-requests — Yet Another HTTP Library for OpenResty

Here, we see 8 third-party libraries for lua-resty-http. Comparing this to the previous results, we found only 2 using OPM, and only 1 on LuaRocks. However, if you have difficulty deciding, you can choose the first one, as it is the same as the one in LuaRocks.

For engineers who are willing to explore, I recommend using the last library, lua-resty-requests. It is a more user-friendly HTTP client library, with an interface style similar to the famous Python library Requests. If, like me, you are a Python enthusiast, you will definitely love lua-resty-requests. The author of this library is tokers, an active member of the OpenResty community, so you can use it with confidence.

We must admit that the existing third-party libraries for OpenResty are not perfect. So, if you cannot find the library you need in awesome-resty, you will need to implement it yourself, such as a lua-resty library for accessing Oracle or SQLServer.

In Conclusion #

For an open-source project to thrive and prosper, it not only needs solid technical foundations, comprehensive documentation, and thorough testing, but also the involvement of more developers and companies to create an ecosystem. As the Apache Foundation wisely notes, the community is more important than the code.

It is no simple task to write quality OpenResty code. There is currently a lack of systematic learning materials and official code guidelines for OpenResty. While many optimization points are documented in the open-source project, most developers understand the end result without fully comprehending the underlying reasons. This is precisely the purpose of this column - I hope that after studying it, you will be able to write more efficient OpenResty code and easily contribute to related open-source projects.

How do you view the OpenResty ecosystem? Feel free to leave a comment and let’s discuss. Also, please share this article with your colleagues and friends so that we can all progress together through exchange of ideas.