24 Web Service Web Service Core Functionality and Implementation

24 Web Service Web Service Core Functionality and Implementation #

Hello, I am Kong Lingfei. Starting today, we will enter the third phase of practical training: service development. In this section, I will explain the construction methods of various services in the IAM project, helping you master various skill points in Go development.

In Go project development, in most cases, we are writing backend services that provide certain functionalities. These functionalities are exposed to the outside world in the form of RPC API interfaces or RESTful API interfaces, and services that can provide these two types of API interfaces are collectively referred to as web services. Today, I will introduce how to implement the core functions of web services by introducing RESTful API-style web services.

So let’s take a look at the core functions of web services and how to develop them today.

Core Functions of a Web Service #

A Web service has many functions, which I have categorized into two main types: basic functions and advanced functions. I have summarized these functions in the following diagram:

Now, I will explain these functions in the order shown in the diagram.

To implement a Web service, first, we need to choose a communication protocol and format. In Go project development, there are two combinations available: HTTP+JSON and gRPC+Protobuf. Since the iam-apiserver mainly provides REST-style API interfaces, we have chosen the HTTP+JSON combination.

The most core function of a Web service is routing matching. Routing matching is essentially matching the (HTTP method, request path) to the function that handles this request. Finally, this function processes the request and returns the result, as shown in the following diagram:

An HTTP request goes through routing matching and is eventually handled by the Delete(c *gin.Context) function. The variable c holds the parameters of this request. In the Delete function, we can perform parameter parsing, parameter validation, logical processing, and finally return the result.

For large-scale systems, there may be many API interfaces, and these interfaces may have multiple versions as requirements are updated and iterated. To facilitate management, we need to group the routes.

Sometimes, we need to simultaneously open ports 80 for HTTP services and 443 for HTTPS services in a single service process. This allows us to simplify internal service access by using port 80, and provide more secure HTTPS services to external clients. Obviously, we don’t need to start multiple service processes for the same functionality. Therefore, the Web service needs to support one process with multiple services.

The most essential requirement for developing a Web service is to input some parameters, validate them, perform business logic processing, and finally return the result. Therefore, the Web service should also be able to perform functions such as parameter parsing, parameter validation, logical processing, and result retrieval. These are all business processing functions of a Web service.

The above functions are the basic functions of a Web service. In addition, we also need to support some advanced functions.

When making an HTTP request, it is often necessary to perform certain common operations for each request, such as adding headers, adding a RequestID, or counting the number of requests. This requires our Web service to support the middleware feature.

To ensure system security, we need to perform authentication for each request. In Web services, there are usually two authentication methods: username and password-based authentication, and token-based authentication. After successful authentication, the request can be processed further.

To facilitate locating and tracking a specific request, we need to support the use of RequestIDs. Locating and tracking RequestIDs are mainly used for troubleshooting.

Finally, in current software architectures, many adopt a front-end and back-end separation architecture. In this architecture, the front-end and back-end may have different access addresses. For security reasons, browsers set cross-origin requests for this situation. Therefore, the Web service needs to be able to handle cross-origin requests from browsers.

So far, I have explained the basic functions and advanced functions of a Web service. Of course, the above only introduces the core functions of a Web service, and there are many other functions that you can learn about by studying the Gin official documentation.

As you can see, a Web service has many core functions that we can implement by encapsulating the net/http package ourselves. However, in actual project development, we are more likely to choose excellent open-source Web frameworks based on the net/http package for encapsulation. This practical project has chosen the Gin framework.

Next, let’s mainly focus on how the Gin framework implements the aforementioned core functions, which we can directly use in actual development.

Why choose the Gin framework? #

There are many excellent web frameworks available, so why should we choose Gin? Before answering this question, let’s take a look at the factors to consider when choosing a web framework.

When selecting a web framework, we can focus on the following points:

  • Routing functionality
  • Whether it has middleware/filter capabilities
  • HTTP parameter (path, query, form, header, body) parsing and response
  • Performance and stability
  • Complexity of use
  • Community activity

Based on the number of GitHub stars, the current popular Go web frameworks include Gin, Beego, Echo, Revel, and Martini. After conducting research, I have chosen the Gin framework for the following reasons:

  • Lightweight with high code quality and performance
  • The project is currently active and has many available middleware
  • As a web framework, it is fully functional and easy to use

Next, I will provide a detailed introduction to the Gin framework.

Gin is a web framework written in Go, which is feature-rich, easy to use, and highly performant. Gin’s core routing functionality is implemented through a customized version of HttpRouter, which provides high routing performance.

Gin has many features, and I’ll list some of its core features for you:

  • Supports HTTP methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
  • Supports different types of HTTP parameters: path, query, form, header, and body
  • Supports HTTP routing and route grouping
  • Supports middleware and custom middleware
  • Supports custom logging
  • Supports binding and validation, and allows custom validators. Can bind parameters such as query, path, body, header, form
  • Supports redirection
  • Supports basic auth middleware
  • Supports custom HTTP configurations
  • Supports graceful shutdown
  • Supports HTTP2
  • Supports setting and getting cookies

How does Gin support basic web service features? #

Next, let’s take a look at how Gin supports basic web service features using a concrete example. We will then provide detailed explanations on how to use these features.

First, let’s create a directory called “webfeature” to store our sample code. Since we will be demonstrating the usage of HTTPS, we need to create certificate files. This process can be divided into two steps.

Step 1: Execute the following command to create the certificate:

cat << 'EOF' > ca.pem
-----BEGIN CERTIFICATE-----
MIICSjCCAbOgAwIBAgIJAJHGGR4dGioHMA0GCSqGSIb3DQEBCwUAMFYxCzAJBgNV
BAYTAkFVMRMwEQYDVQQIEwpTb21lLVN0YXRlMSEwHwYDVQQKExhJbnRlcm5ldCBX
aWRnaXRzIFB0eSBMdGQxDzANBgNVBAMTBnRlc3RjYTAeFw0xNDExMTEyMjMxMjla
Fw0yNDExMDgyMjMxMjlaMFYxCzAJBgNVBAYTAkFVMRMwEQYDVQQIEwpTb21lLVN0
YXRlMSEwHwYDVQQKExhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQxDzANBgNVBAMT
BnRlc3RjYTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAwEDfBV5MYdlHVHJ7
+L4nxrZy7mBfAVXpOc5vMYztssUI7mL2/iYujiIXM+weZYNTEpLdjyJdu7R5gGUu
g1jSVK/EPHfc74O7AyZU34PNIP4Sh33N+/A5YexrNgJlPY+E3GdVYi4ldWJjgkAd
Qah2PH5ACLrIIC6tRka9hcaBlIECAwEAAaMgMB4wDAYDVR0TBAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAgQwDQYJKoZIhvcNAQELBQADgYEAHzC7jdYlzAVmddi/gdAeKPau
sPBG/C2HCWqHzpCUHcKuvMzDVkY/MP2o6JIW2DBbY64bO/FceExhjcykgaYtCH/m
oIU63+CFOTtR7otyQAWHqXa7q4SbCDlG7DyRFxqG0txPtGvy12lgldA2+RgcigQG
Dfcog5wrJytaQ6UA0wE=
-----END CERTIFICATE-----
EOF

cat << 'EOF' > server.key
-----BEGIN PRIVATE KEY-----
MIICdQIBADANBgkqhkiG9w0BAQEFAASCAl8wggJbAgEAAoGBAOHDFScoLCVJpYDD
M4HYtIdV6Ake/sMNaaKdODjDMsux/4tDydlumN+fm+AjPEK5GHhGn1BgzkWF+slf
3BxhrA/8dNsnunstVA7ZBgA/5qQxMfGAq4wHNVX77fBZOgp9VlSMVfyd9N8YwbBY
AckOeUQadTi2X1S6OgJXgQ0m3MWhAgMBAAECgYAn7qGnM2vbjJNBm0VZCkOkTIWm
V10okw7EPJrdL2mkre9NasghNXbE1y5zDshx5Nt3KsazKOxTT8d0Jwh/3KbaN+YY
tTCbKGW0pXDRBhwUHRcuRzScjli8Rih5UOCiZkhefUTcRb6xIhZJuQy71tjaSy0p
dHZRmYyBYO2YEQ8xoQJBAPrJPhMBkzmEYFtyIEqAxQ/o/A6E+E4w8i+KM7nQCK7q
K4JXzyXVAjLfyBZWHGM2uro/fjqPggGD6QH1qXCkI4MCQQDmdKeb2TrKRh5BY1LR
81aJGKcJ2XbcDu6wMZK4oqWbTX2KiYn9GB0woM6nSr/Y6iy1u145YzYxEV/iMwff
DJULAkB8B2MnyzOg0pNFJqBJuH29bKCcHa8gHJzqXhNO5lAlEbMK95p/P2Wi+4Hd
aiEIAF1BF326QJcvYKmwSmrORp85AkAlSNxRJ50OWrfMZnBgzVjDx3xG6KsFQVk2
ol6VhqL6dFgKUORFUWBvnKSyhjJxurlPEahV6oo6+A+mPhFY8eUvAkAZQyTdupP3
XEFQKctGz+9+gKkemDp7LBBMEMBXrGTLPhpEfcjv/7KPdnFHYmhYeBTBnuVmTVWe
F98XJ7tIFfJq
-----END PRIVATE KEY-----
EOF

cat << 'EOF' > server.pem
-----BEGIN CERTIFICATE-----
MIICnDCCAgWgAwIBAgIBBzANBgkqhkiG9w0BAQsFADBWMQswCQYDVQQGEwJBVTET
MBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0cyBQ
dHkgTHRkMQ8wDQYDVQQDEwZ0ZXN0Y2EwHhcNMTUxMTA0MDIyMDI0WhcNMjUxMTAx
MDIyMDI0WjBlMQswCQYDVQQGEwJVUzERMA8GA1UECBMISWxsaW5vaXMxEDAOBgNV
BAcTB0NoaWNhZ28xFTATBgNVBAoTDEV4YW1wbGUsIENvLjEaMBgGA1UEAxQRKi50
ZXN0Lmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAOHDFSco
LCVJpYDDM4HYtIdV6Ake/sMNaaKdODjDMsux/4tDydlumN+fm+AjPEK5GHhGn1Bg
zkWF+slf3BxhrA/8dNsnunstVA7ZBgA/5qQxMfGAq4wHNVX77fBZOgp9VlSMVfyd

9N8YwbBYAckOeUQadTi2X1S6OgJXgQ0m3MWhAgMBAAGjazBpMAkGA1UdEwQCMAAw
CwYDVR0PBAQDAgXgME8GA1UdEQRIMEaCECoudGVzdC5nb29nbGUuZnKCGHdhdGVy
em9vaS50ZXN0Lmdvb2dsZS5iZYISKi50ZXN0LnlvdXR1YmUuY29thwTAqAEDMA0G
CSqGSIb3DQEBCwUAA4GBAJFXVifQNub1LUP4JlnX5lXNlo8FxZ2a12AFQs+bzoJ6
hM044EDjqyxUqSbVePK0ni3w1fHQB5rY9yYC5f8G7aqqTY1QOhoUk8ZTSTRpnkTh
y4jjdvTZeLDVBlueZUTDRmy2feY5aZIU18vFDK08dTG0A87pppuv1LNIR3loveU8
-----END CERTIFICATE-----
EOF

Second, create the main.go file:

package main

import (
	"fmt"
	"log"
	"net/http"
	"sync"
	"time"

	"github.com/gin-gonic/gin"
	"golang.org/x/sync/errgroup"
)

type Product struct {
	Username    string    `json:"username" binding:"required"`
	Name        string    `json:"name" binding:"required"`
	Category    string    `json:"category" binding:"required"`
	Price       int       `json:"price" binding:"gte=0"`
	Description string    `json:"description"`
	CreatedAt   time.Time `json:"createdAt"`
}

type productHandler struct {
	sync.RWMutex
	products map[string]Product
}

func newProductHandler() *productHandler {
	return &productHandler{
		products: make(map[string]Product),
	}
}

func (u *productHandler) Create(c *gin.Context) {
	u.Lock()
	defer u.Unlock()

	// 1. Parameter parsing
	var product Product
	if err := c.ShouldBindJSON(&product); err != nil {
		c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
		return
	}

insecureServer := &http.Server{
    Addr:         ":8080",
    Handler:      router(),
    ReadTimeout:  5 * time.Second,
    WriteTimeout: 10 * time.Second,
}

这里创建了一个http.Server实例insecureServer，并指定了监听地址为:8080，Handler为router()函数返回的路由器。同时设置了ReadTimeout和WriteTimeout。然后通过调用ListenAndServe()方法启动HTTP服务。

要启动一个HTTPS服务，只需要创建一个http.Server实例并指定TLS证书和密钥的路径，如下所示：

secureServer := &http.Server{
    Addr:         ":8443",
    Handler:      router(),
    ReadTimeout:  5 * time.Second,
    WriteTimeout: 10 * time.Second,
}

err := secureServer.ListenAndServeTLS("server.pem", "server.key")
if err != nil && err != http.ErrServerClosed {
    log.Fatal(err)
}

这里创建了一个http.Server实例secureServer，并指定了监听地址为:8443，Handler同样是router()函数返回的路由器。然后通过调用ListenAndServeTLS()方法启动HTTPS服务，并指定TLS证书和密钥的路径。

路由注册与分组 #

在Gin中，可以通过路由注册和分组的方式来定义不同的URL路径和处理函数。

productv1 := v1.Group("/products")
{
    // 路由匹配
    productv1.POST("", productHandler.Create)
    productv1.GET(":name", productHandler.Get)
}

以上代码中，首先通过Group("/products")方法创建了一个名为productv1的路由分组，指定了URL前缀为/products。然后在这个路由分组中注册了两个路由，分别是POST请求和GET请求。POST请求的处理函数是productHandler.Create，GET请求的处理函数是productHandler.Get。

路由的注册方法有很多，比如常见的GET、POST、PUT、DELETE等，还有Handle()方法用于注册任何请求方法。

请求和响应处理 #

在处理HTTP请求时，可以通过c *gin.Context参数获取HTTP请求的相关信息，并通过其提供的方法来处理请求和生成响应。

func (u *productHandler) Create(c *gin.Context) {
    // 参数校验
    if _, ok := u.products[product.Name]; ok {
        c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("product %s already exist", product.Name)})
        return
    }

    product.CreatedAt = time.Now()

    // 逻辑处理
    u.products[product.Name] = product
    log.Printf("Register product %s success", product.Name)

    // 返回结果
    c.JSON(http.StatusOK, product)
}

以上代码是productHandler结构体的Create方法的实现，用于处理创建产品的请求。在这个方法中，首先进行参数校验，如果产品已经存在，则返回错误信息；接着为产品设置创建时间；然后进行实际的逻辑处理，将该产品添加到products映射中，并打印日志；最后通过调用JSON()方法将产品数据以JSON格式返回给客户端。

类似地，Get方法用于处理获取产品信息的请求，其实现与Create方法类似。

Gin提供了丰富的内置方法来处理请求和生成响应，比如Query()、Param()、JSON()、XML()、HTML()等。

中间件支持 #

Gin支持中间件，可以通过中间件在请求和响应中添加额外的逻辑和处理。在路由注册时，可以使用Use()方法添加中间件。

router := gin.Default()
productHandler := newProductHandler()
// 路由分组、中间件、认证
v1 := router.Group("/v1", AuthMiddleware())
{
    productv1 := v1.Group("/products")
    {
        // 路由匹配
        productv1.POST("", productHandler.Create)
        productv1.GET(":name", productHandler.Get)
    }
}

以上代码中，通过调用AuthMiddleware()函数创建了一个认证中间件，并通过Use()方法添加到路由分组v1上。

中间件可以在请求前、请求后、响应前或响应后执行一些额外的逻辑操作，比如认证、日志、跨域处理、压缩等。

错误处理 #

Gin提供了一种方式来处理错误，即在处理函数中返回一个error类型的值。可以将错误信息作为响应内容返回给客户端。

if !ok {
    c.JSON(http.StatusNotFound, gin.H{"error": fmt.Errorf("can not found product %s", c.Param("name"))})
    return
}

以上代码中，如果未找到产品，则返回一个带有错误信息的StatusNotFound响应。

总结 #

通过以上示例代码，我们了解了Gin框架如何支持Web基础功能，包括HTTP/HTTPS支持、路由注册与分组、请求和响应处理、中间件支持、错误处理等。在实际开发中，可以根据需要选择适合的功能和组件来构建Web服务。

insecureServer := &http.Server{
    Addr:         ":8080",
    Handler:      router(),
    ReadTimeout:  5 * time.Second,
    WriteTimeout: 10 * time.Second,
}
...
err := insecureServer.ListenAndServe()

To start an HTTPS server, use the following code:

secureServer := &http.Server{
    Addr:         ":8443",
    Handler:      router(),
    ReadTimeout:  5 * time.Second,
    WriteTimeout: 10 * time.Second,
}
...
err := secureServer.ListenAndServeTLS("server.pem", "server.key")

JSON Data Format Support #

Gin supports various data communication formats such as application/json and application/xml. You can use the c.ShouldBindJSON function to parse JSON data in the request body into a specified struct, and use the c.JSON function to return JSON formatted data.

Routing Matching #

Gin supports two types of routing matching rules.

The first type of matching rule is exact matching. For example, with a route of /products/:name, the matching cases are shown in the table below:

/products/iphone      # matches "/products/:name"
/products/macbook     # matches "/products/:name"
/products/ipad        # matches "/products/:name"

The second type of matching rule is wildcard matching. For example, with a route of /products/*name, the matching cases are shown in the table below:

/products                 # doesn't match "/products/*name"
/products/iphone           # matches "/products/*name"
/products/macbook/pro      # matches "/products/*name"
/products/ipad/am          # matches "/products/*name"

Routing Groups #

Gin provides routing grouping functionality through the Group function. Routing grouping is a commonly used feature that allows you to group routes of the same version or routes for the same RESTful resource. For example:

v1 := router.Group("/v1", gin.BasicAuth(gin.Accounts{"foo": "bar", "colin": "colin404"}))
{
    productv1 := v1.Group("/products")
    {
        // route matching
        productv1.POST("", productHandler.Create)
        productv1.GET(":name", productHandler.Get)
    }

    orderv1 := v1.Group("/orders")
    {
        // route matching
        orderv1.POST("", orderHandler.Create)
        orderv1.GET(":name", orderHandler.Get)
    }
}

v2 := router.Group("/v2", gin.BasicAuth(gin.Accounts{"foo": "bar", "colin": "colin404"}))
{
    productv2 := v2.Group("/products")
    {
        // route matching
        productv2.POST("", productHandler.Create)
        productv2.GET(":name", productHandler.Get)
    }
}

By grouping the routes, you can apply uniform processing to routes of the same group. For example, in the above example, by using the code v1 := router.Group("/v1", gin.BasicAuth(gin.Accounts{"foo": "bar", "colin": "colin404"})), you can add the gin.BasicAuth middleware to all routes in the v1 group to implement authentication. We won’t go into middleware and authentication here, but we will discuss them later in the advanced features section.

Multiple Services in One Process #

You can implement multiple services in a single process using the following code:

var eg errgroup.Group
insecureServer := &http.Server{...}
secureServer := &http.Server{...}

eg.Go(func() error {
    err := insecureServer.ListenAndServe()
    if err != nil && err != http.ErrServerClosed {
        log.Fatal(err)
    }
    return err
})
eg.Go(func() error {
    err := secureServer.ListenAndServeTLS("server.pem", "server.key")
    if err != nil && err != http.ErrServerClosed {
        log.Fatal(err)
    }
    return err
}

if err := eg.Wait(); err != nil {
    log.Fatal(err)
})

The above code implements two identical services listening on different ports. To prevent blocking the start of the second service, we need to execute the ListenAndServe function in a goroutine and use eg.Wait() to block the program process, allowing both HTTP services to continuously listen to ports and serve requests in goroutines.

Parameter Parsing, Parameter Validation, Logic Processing, Returning Result #

In addition, a web service should also have four functions: parameter parsing, parameter validation, logic processing, and returning results. These functions are closely related and will be discussed together.

In the Create method of the productHandler, we use c.ShouldBindJSON to parse the parameters. Then we write our own validation code and save the product information in memory (i.e., business logic processing). Finally, we return the created product information through c.JSON. Here is the code:

func (u *productHandler) Create(c *gin.Context) {
    u.Lock()
    defer u.Unlock()

    // 1. Parameter Parsing
    var product Product
    if err := c.ShouldBindJSON(&product); err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    // 2. Parameter Validation
    if _, ok := u.products[product.Name]; ok {
        c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("product %s already exist", product.Name)})
        return
    }
    product.CreatedAt = time.Now()

    // 3. Logic Processing
    u.products[product.Name] = product
    log.Printf("Register product %s success", product.Name)

    // 4. Returning Result
    c.JSON(http.StatusOK, product)
}

Now you might ask: HTTP request parameters can exist in different locations, how does Gin parse them? Let’s take a look at the different types of HTTP parameters. There are 5 types of HTTP parameters:

  • Path parameters. For example, gin.Default().GET("/user/:name", nil), where name is a path parameter.
  • Query string parameters. For example, /welcome?firstname=Lingfei&lastname=Kong, where firstname and lastname are query string parameters.
  • Form parameters. For example, curl -X POST -F 'username=colin' -F 'password=colin1234' http://mydomain.com/login, where username and password are form parameters.
  • HTTP header parameters. For example, curl -X POST -H 'Content-Type: application/json' -d '{"username":"colin","password":"colin1234"}' http://mydomain.com/login, where Content-Type is an HTTP header parameter.
  • Body parameters. For example, curl -X POST -H 'Content-Type: application/json' -d '{"username":"colin","password":"colin1234"}' http://mydomain.com/login, where username and password are body parameters.

Gin provides some functions to read these HTTP parameters. For each category, there are two types of functions available: one type can directly read the value of a specific parameter, and the other type can bind parameters of the same category into a Go structure. For example, if we have the following path parameters:

gin.Default().GET("/:name/:id", nil)

We can directly read each parameter:

name := c.Param("name")
id := c.Param("id")

Or we can bind all path parameters to a structure:

type Person struct {
    ID   string `uri:"id" binding:"required,uuid"`
    Name string `uri:"name" binding:"required"`
}

if err := c.ShouldBindUri(&person); err != nil {
    // handle error
    return
}

When binding parameters, Gin uses tags in the structure to determine which type of parameters should be bound. It’s worth noting that different types of parameters have different tags.

  • Path parameters: uri.
  • Query string parameters: form.
  • Form parameters: form.
  • HTTP header parameters: header.
  • Body parameters: The binding engine will automatically choose between JSON or XML based on the Content-Type. You can also use ShouldBindJSON or ShouldBindXML to explicitly specify the tag to use.

For each category of parameters, Gin provides corresponding functions to get and bind these parameters. These functions are based on the following two functions:

  1. ShouldBindWith(obj interface{}, b binding.Binding) error

This is an important function. Many ShouldBindXXX functions underlyingly call ShouldBindWith to perform parameter binding. This function binds the parameters to the given structure pointer based on the binding engine provided. If the binding fails, only the error content is returned without terminating the HTTP request. ShouldBindWith supports multiple binding engines, such as binding.JSON, binding.Query, binding.Uri, binding.Header, etc. For more detailed information, you can refer to binding.go.

  1. MustBindWith(obj interface{}, b binding.Binding) error

This is another important function. Many BindXXX functions underlyingly call MustBindWith to perform parameter binding. This function binds the parameters to the given structure pointer based on the binding engine provided. If the binding fails, an error is returned and the request is terminated with an HTTP 400 error. MustBindWith supports the same binding engines as ShouldBindWith.

Based on these two functions, Gin has derived many new Bind functions to meet the needs of different scenarios for obtaining HTTP parameters. Gin provides functions to get parameters from 5 categories:

  • Path parameters: ShouldBindUri, BindUri
  • Query string parameters: ShouldBindQuery, BindQuery
  • Form parameters: ShouldBind
  • HTTP header parameters: ShouldBindHeader, BindHeader
  • Body parameters: ShouldBindJSON, BindJSON, etc.

For each category, there are multiple Bind functions. For more information, you can refer to the Bind functions provided by Gin.

It’s worth noting that Gin does not provide functions like ShouldBindForm or BindForm to bind form parameters specifically. However, we can use ShouldBind to bind form parameters. When the HTTP method is GET, ShouldBind only binds parameters of the Query type. When the HTTP method is POST, it first checks whether the content-type is JSON or XML. If it’s not, it binds parameters of the Form type.

Therefore, ShouldBind can bind form parameters, but it requires the HTTP method to be POST and the content-type to not be application/json or application/xml.

In Go project development, I recommend using ShouldBindXXX so that we can ensure that the HTTP chain (chain can be understood as a series of processing plugins for an HTTP request) we set can continue to be executed.

How does Gin support advanced features for web services? #

In the previous section, we introduced the basic features of web services. Now let’s talk about the advanced features. Web services can have multiple advanced features, but the core ones are middleware, authentication, RequestID, cross-origin resource sharing (CORS), and graceful shutdown.

Middleware #

Gin supports middleware, which allows HTTP requests to be processed by a series of loaded middleware before being forwarded to the actual handler functions. In middleware, you can parse the HTTP requests and perform some logical operations, such as handling CORS or generating an X-Request-ID and saving it in the context for request tracing. After processing, you can choose to interrupt and return the request or continue passing it to the next middleware. Only after all the middleware have been processed, the request will be handed over to the routing function for further processing. The specific workflow is shown in the following diagram:

By using middleware, you can implement unified processing for all requests, improve development efficiency, and make your code more concise. However, because all requests need to go through middleware processing, it may increase request latency. Here are some suggestions for using middleware:

  • Make middleware loadable, allowing the configuration file to specify which middleware to load when the program starts.
  • Only make essential and commonly used functionalities as middleware.
  • When writing middleware, ensure the code quality and performance of the middleware.

In Gin, you can load middleware using the Use method of gin.Engine. Middleware can be loaded at different positions, and the scope of their effects also varies. For example:

router := gin.New()
router.Use(gin.Logger(), gin.Recovery()) // Middleware applies to all HTTP requests
v1 := router.Group("/v1").Use(gin.BasicAuth(gin.Accounts{"foo": "bar", "colin": "colin404"})) // Middleware applies to the v1 group
v1.POST("/login", Login).Use(gin.BasicAuth(gin.Accounts{"foo": "bar", "colin": "colin404"})) // Middleware applies only to the /v1/login API endpoint

The Gin framework itself provides built-in middlewares:

  • gin.Logger(): The Logger middleware writes logs to gin.DefaultWriter, which defaults to os.Stdout.
  • gin.Recovery(): The Recovery middleware recovers from any panic and writes a 500 status code.
  • gin.CustomRecovery(handle gin.RecoveryFunc): Similar to the Recovery middleware, but also calls the handle function for processing during recovery.
  • gin.BasicAuth(): HTTP basic authentication (authentication using usernames and passwords).

Furthermore, Gin supports custom middleware. Middleware is essentially a function with the type gin.HandlerFunc, and the underlying type of HandlerFunc is func(*Context). Here is an example implementation of a Logger middleware:

package main

import (
	"log"
	"time"

	"github.com/gin-gonic/gin"
)

func Logger() gin.HandlerFunc {
	return func(c *gin.Context) {
		t := time.Now()

		// Set a variable 'example'
		c.Set("example", "12345")

		// Before request

		c.Next()

		// After request
		latency := time.Since(t)
		log.Print(latency)

		// Access the status we are sending
		status := c.Writer.Status()
		log.Println(status)
	}
}

func main() {
	r := gin.New()
	r.Use(Logger())

	r.GET("/test", func(c *gin.Context) {
		example := c.MustGet("example").(string)

		// It would print: "12345"
		log.Println(example)
	})

	// Listen and serve on 0.0.0.0:8080
	r.Run(":8080")
}

There are also many open-source middleware options available for us to choose from. I have summarized some commonly used ones in the following table:

Authentication, RequestID, CORS #

Authentication, RequestID, and CORS (cross-origin resource sharing) can all be implemented using middleware in Gin. For example:

router := gin.New()

// Authentication
router.Use(gin.BasicAuth(gin.Accounts{"foo": "bar", "colin": "colin404"}))

// RequestID
router.Use(requestid.New(requestid.Config{
	Generator: func() string {
		return "test"
	},
	// ...
}))

}))

// Cross-domain
// CORS for https://foo.com and https://github.com origins, allowing:
// - PUT and PATCH methods
// - Origin header
// - Credentials share
// - Preflight requests cached for 12 hours
router.Use(cors.New(cors.Config{
    AllowOrigins:     []string{"https://foo.com"},
    AllowMethods:     []string{"PUT", "PATCH"},
    AllowHeaders:     []string{"Origin"},
    ExposeHeaders:    []string{"Content-Length"},
    AllowCredentials: true,
    AllowOriginFunc: func(origin string) bool {
        return origin == "https://github.com"
    },
    MaxAge: 12 * time.Hour,
}))

Graceful Shutdown #

After a Go project is deployed, we need to continuously iterate to enrich the project with new features, fix bugs, etc. This also means that we need to restart the Go service constantly. For an HTTP service, if there is high traffic, there may still be many connections that have not been closed and requests that have not been completed when restarting the service. If we directly close the service at this time, these connections will be terminated directly, causing requests to be terminated abnormally, which can have a significant impact on user experience and product reputation. Therefore, this kind of closing method is not an elegant way to shut down.

At this point, we expect the HTTP service to gracefully close these connections after processing all requests, which means closing the service gracefully. There are two methods for gracefully shutting down an HTTP service, using a third-party Go package or implementing it yourself.

Method 1: Using a third-party Go package

If we use a third-party Go package to achieve graceful shutdown, the package that is currently used more frequently is fvbock/endless. We can use fvbock/endless to replace the ListenAndServe method of net/http, for example:

router := gin.Default()
router.GET("/", handler)
// [...]
endless.ListenAndServe(":4242", router)

Method 2: Implementation on your own

The advantage of using a third-party package is that it can slightly reduce the coding workload, but the disadvantage is that it introduces a new dependency package. Therefore, I prefer to implement it myself. Starting from Go version 1.8 or later, the Shutdown method built into http.Server has implemented graceful shutdown. Here is an example:

// +build go1.8

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/gin-gonic/gin"
)

func main() {
	router := gin.Default()
	router.GET("/", func(c *gin.Context) {
		time.Sleep(5 * time.Second)
		c.String(http.StatusOK, "Welcome Gin Server")
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: router,
	}

	// Initializing the server in a goroutine so that
	// it won't block the graceful shutdown handling below
	go func() {
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatalf("listen: %s\n", err)
		}
	}()

	// Wait for interrupt signal to gracefully shutdown the server with
	// a timeout of 5 seconds.
	quit := make(chan os.Signal, 1)
	// kill (no param) default send syscall.SIGTERM
	// kill -2 is syscall.SIGINT
	// kill -9 is syscall.SIGKILL but can't be catch, so don't need add it
	signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
	<-quit
	log.Println("Shutting down server...")

	// The context is used to inform the server it has 5 seconds to finish
	// the request it is currently handling
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	if err := srv.Shutdown(ctx); err != nil {
		log.Fatal("Server forced to shutdown:", err)
	}

	log.Println("Server exiting")
}

In the example above, srv.ListenAndServe needs to be executed in a goroutine so that it does not block the srv.Shutdown function. Because we put srv.ListenAndServe in a goroutine, we need a mechanism to keep the entire process running.

Here, we use a buffered channel and call the signal.Notify function to bind the channel to the SIGINT and SIGTERM signals. In this way, when the SIGINT or SIGTERM signal is received, the quit channel will be written with a value, which will end the blocking state and allow the program to continue running and execute srv.Shutdown(ctx), gracefully shutting down the HTTP service.

Summary #

Today we mainly learned about the core functionalities of web services and how to develop them. In practical project development, we generally use excellent open-source web frameworks based on the net/http package.

Currently, popular Go web frameworks include Gin, Beego, Echo, Revel, and Martini. You can choose according to your needs. I personally recommend Gin, which is also a popular web framework currently. Gin web framework supports many basic functionalities of web services, such as HTTP/HTTPS, JSON formatted data, routing grouping and matching, and one process serving multiple services.

Additionally, Gin also supports some advanced functionalities of web services, such as middleware, authentication, RequestID, cross-origin resource sharing (CORS), and graceful shutdown.

Exercise after class #

  1. Use the Gin framework to write a simple web service. The web service should be able to parse and validate parameters, perform some simple business logic, and finally return the processing result. Feel free to share your achievements or any problems you encounter in the comments section.
  2. Consider how to add a rate limiting middleware to the /healthz interface of iam-apiserver to limit the frequency of requests to /healthz.

Feel free to leave a comment to discuss and communicate with me. See you in the next lecture.