27 How Should the API Gateway System Gateway Be Done #
Hello, I’m Tang Yang.
So far, your vertical e-commerce system has been running for a while after being split into microservices. The system’s scalability has significantly improved, and it can handle peak traffic more smoothly.
However, recently you have noticed that as your e-commerce website becomes more popular, the system has attracted some “uninvited guests.” During the early hours of the morning, there is a dramatic increase in the number of calls to the search products and user interfaces, which lasts for a period of time and then returns to normal.
These search requests share a common feature—they come from a few fixed devices. After adding rate limiting based on device ID to the search service, the peak search requests during the early hours disappear. However, not long after, there is a large number of requests to crawl user information in the user service and crawl product information in the product interface. You have to add the same rate limiting strategy to these two services as well.
However, this introduces a problem: using the same strategy on three different services results in redundant code and makes code reuse impossible. If similar issues arise in other services, you would have to copy the code to implement it, which is clearly not a viable solution.
As a Java programmer, you quickly come up with an idea: separate the rate limiting functionality into a standalone jar package for the three services to reference. However, you overlooked a scenario where your e-commerce team uses not only Java but also other languages like PHP and Golang.
For services developed in multiple languages, using a jar package to implement rate limiting functionality is not feasible. This is where you need to introduce an API Gateway.
The Role of API Gateway (904) #
API Gateway is not an open-source component, but rather an architectural pattern. It integrates common functionalities of services together and deploys them as a separate layer to solve governance problems of services. You can think of it as the system boundary, which provides unified control over the inflow and outflow of system traffic.
In my opinion, API Gateway can be divided into two categories: entrance gateway and exit gateway.
The entrance gateway is a commonly used type of gateway. It is deployed between the load balancer server and the application server, and it has several key functions.
Firstly, it provides clients with a uniform access address. API Gateway can dynamically route clients’ requests to different business services and perform necessary protocol conversion. In your system, the protocols exposed by your deployed microservices may vary: some provide HTTP services, some have been transformed into RPC and provide RPC services, and some legacy systems may still expose Web Service services. API Gateway can shield the clients from these service deployment addresses and protocol details, greatly facilitating the clients’ calls.
Secondly, in API Gateway, we can implement various service governance strategies, such as service circuit breaking, degradation, traffic control, and routing (I will explain the details of service degradation and traffic control in later courses; for now, you just need to know that they can be implemented in API Gateway).
In addition, the implementation of client authentication and authorization can also be placed in API Gateway. You should understand that different types of clients use different authentication methods. In my previous project, mobile apps used OAuth protocol for authentication, HTML5 and web clients used cookie authentication, and internal services used a self-developed token authentication method. These authentication methods can be uniformly handled in API Gateway, and application services do not need to understand the authentication details.
Furthermore, API Gateway can also handle blacklists and whitelists related to device IDs, user IPs, user IDs, etc.
Lastly, API Gateway can also perform log recording tasks, such as recording access logs of HTTP requests. As I mentioned in Lesson 25 when discussing distributed tracing systems, it is possible to generate a request ID in the gateway.
Exit gateway does not have such rich functionalities and roles. In system development, we rely on many external third-party systems, such as typical examples: third-party account login, using third-party payment tools, etc. We can deploy an exit gateway between the application server and the third-party system to provide unified authentication, authorization, auditing, and access control for calling external APIs.
I have spent a certain amount of space to explain the role of API Gateway mainly to let you understand what practical problems API Gateway can solve. This way, when you face these problems, you will have a solution in mind and will not be at a loss.
How to Implement an API Gateway #
After understanding the role of an API gateway, let’s take a look at what points need to be considered when implementing an API gateway and what common open-source API gateways are available. This way, whether you are considering building your own API gateway or using an open-source implementation, you will be able to work with it more smoothly.
When implementing an API gateway, the first thing you need to consider is its performance. This is easy to understand, as the API gateway handles all the traffic from clients. For example, if the processing time for a business service is 10ms, and the API gateway takes 1ms, then the response time for each interface will increase by 10%. This undoubtedly has a huge impact on performance. The key to improving API gateway performance lies in the I/O model (which I discussed in detail in Lecture 23). Here, I’ll just give an example to illustrate the impact of the I/O model on performance.
In the 1.0 version of the API gateway Zuul, which is open sourced by Netflix, a synchronous blocking I/O model was used. The overall system is actually a servlet that receives requests from users, executes authentication, protocol conversion, and other logic configured in the gateway, and then calls backend services to retrieve data to return to the user.
In Zuul 2.0, the Netflix team transformed the servlet into a Netty server (a Netty service), which uses the I/O multiplexing model to handle incoming I/O requests. They also transformed the previous synchronous blocking calls to backend services into non-blocking calls using a Netty client. After the transformation, the Netflix team found that performance improved by about 20%.
In addition, some actions performed in an API gateway can be predefined, such as setting blacklists and whitelists, and dynamic routing of interfaces, while others need to be defined by business units. Therefore, the design of the API gateway should pay attention to scalability, which means you can add or remove logic at any time in the execution chain of the gateway (also called hot swapping).
In general, each operation can be defined as a filter, and these filters can be connected using the “chain of responsibility” pattern. This allows the dynamic organization of these filters, decoupling the relationship between filters. Adding or removing filters will not affect other filters.
Zuul adopts the chain of responsibility pattern. In Zuul 1, filters are classified into three types: pre routing filters, routing filters, and post routing filters. Each filter defines its execution order, and when registering the filter, it will be inserted into the filter chain in the specified order. Therefore, when Zuul receives a request, it will execute the filters inserted into the filter chain in order.
Another point to note is that in order to improve the gateway’s ability to handle requests in parallel, we generally use thread pools to execute requests in parallel. However, this brings a problem: if the product service becomes slow in responding, causing threads calling the product service to be blocked and unable to be released, over time, the threads in the thread pool will be occupied by the product service, thus affecting other services in a cascading manner. Therefore, we need to implement thread isolation or protection for different services. In my opinion, there are two approaches:
If you don’t have many backend services, you can use different thread pools for different services, so that the failure of the product service will not affect the payment service and user service.
Within the thread pool, thread protection can be implemented for different services or even different interfaces. For example, if the maximum number of threads in the thread pool is 1000, a quota can be set for each service on the maximum number of threads it can use.
In general, the execution time for services should be in the millisecond range, and threads should be released quickly after being used, returning to the thread pool for subsequent requests. At the same time, the number of threads in execution should not be too high. By setting quotas on threads for services or interfaces, normal execution will not be affected. However, once a failure occurs and the response time for a certain interface or service becomes longer, causing the number of threads to skyrocket, the quotas will limit the impact on other interfaces or services.
In practice, you can also combine these two approaches. For example, use different thread pools for different services and set quotas for different interfaces within the thread pool.
These are some key points when implementing an API gateway that you can refer to if you want to build your own API gateway service. In addition, there are many open-source implementations of API gateways. Currently, the following are widely used:
Kong is a Lua program running in Nginx. Thanks to Nginx’s performance advantages, Kong is the most efficient in terms of performance compared to other open-source API gateways. As large and medium-sized companies generally have strong Nginx operations and maintenance capabilities, choosing Kong as the API gateway is a good choice in terms of both performance and operational control.
Zuul is a member of the Spring Cloud family. If you are already using other components of Spring Cloud, you can consider using Zuul and integrating it seamlessly with them. However, Zuul 1 is not very efficient in terms of performance because it uses a synchronous blocking model, and Zuul 2 is relatively new and may have some pitfalls. But the code of Zuul is simple and easy to understand, which can be well managed. If your system’s traffic is not as high as Netflix’s, using Zuul is also a good choice, especially for teams using the Java technology stack.
Tyk is a lightweight API gateway implemented in Go language and has a rich set of plug-ins. For teams using the Go language stack, it is also a good choice.
What you need to consider is whether these open-source projects are suitable for your own use as an API gateway. Next, I will use an e-commerce system as an example to guide you in introducing an API gateway into our system.
How to introduce an API gateway into your system? #
So far, our e-commerce system has undergone a service-oriented transformation, with a thin layer of web layer between the service layer and the client. The web layer mainly performs two tasks:
On one hand, it aggregates data from the service layer interfaces. For example, the product details page interface may aggregate data from multiple service interfaces, such as retrieving product information, user information, store information, and user comments.
On the other hand, the web layer also needs to convert HTTP requests into RPC requests and impose some restrictions on frontend traffic, such as adding device IDs to a blacklist.
Therefore, when doing the transformation, we can first separate the API gateway from the web layer and move the protocol conversion, rate limiting, blacklisting, and other tasks to the API gateway to form an independent entry gateway layer.
As for the operation of aggregating service interface data, there are generally two approaches:
Set up a separate gateway for service aggregation and timeout control. The former is often referred to as a traffic gateway, and the latter can be called a business gateway.
Extract a separate service layer dedicated to interface aggregation operations. In this way, the service layer can be roughly divided into atomic service layer and aggregation service layer.
I believe that interface data aggregation is a business operation. Instead of implementing it in a general-purpose gateway layer, it is better to implement it in a service layer closer to the business. Therefore, I am more inclined to the second approach.
At the same time, we can deploy an outbound gateway service between our system and third-party payment services, as well as login services. Previously, you would perform encryption, signing, and other operations on the required data for third-party payment interfaces within the separated payment service, and then call the third-party payment interface to complete the payment request. Now, you can move the encryption and signing operations to the outbound gateway. This way, the payment service only needs to call the unified payment interface of the outbound gateway.
After introducing the API gateway, our system architecture becomes as follows:
Summary of the Course #
In this lesson, I have introduced you to the role of an API gateway in a system, some key points in its implementation, and how to incorporate an API gateway into your system. The key points I want to emphasize are as follows:
API gateways can be divided into entry gateways and exit gateways. Entry gateways have various functions, such as isolating clients and microservices, providing protocol conversion, security policies, authentication, rate limiting, and circuit breaking. Exit gateways mainly provide a unified exit for calling third-party services, where unified authentication, authorization, auditing, and access control for external API calls can be performed;
The implementation of an API gateway focuses on performance and scalability. You can use multi-I/O multiplexing models and thread pool concurrency processing to improve gateway performance. The responsibility chain design pattern can be used to improve the scalability of the gateway;
The thread pool in an API gateway can be used to isolate and protect different interfaces or services, thereby improving the availability of the gateway;
An API gateway can replace the original web layer in a system and move the functions of protocol conversion, authentication, and rate limiting from the web layer to the API gateway, sinking the logic of service aggregation to the service layer.
An API gateway can provide convenience for API calls and can also extract some service governance functions for reuse. Although there may be some performance loss, generally speaking, these losses are acceptable when using mature open-source API gateway components. Therefore, when your microservice system becomes more complex, you can consider using an API gateway as the facade of the entire system.