29 Control Flow Implementing Core Functionality of Iamapiserver Service and Discussion

29 Control Flow Implementing Core Functionality of iamapiserver Service and Discussion #

Hello, I am Kong Lingfei.

In the previous article, I introduced how iam-apiserver builds web services. In this article, let’s take a look at the core features implemented in iam-apiserver. During the explanation of these core features, I will convey my program design ideas to you.

iam-apiserver contains many excellent design ideas and implementations, which may seem fragmented, but I think they are worth sharing with you. I divide these key code designs into three categories: application framework-related features, programming specification-related features, and other features. Next, let’s take a closer look at these design points and the design ideas behind them.

There are three features related to application frameworks, namely graceful shutdown, health check, and plugin-based middleware loading.

Graceful Shutdown #

Before discussing graceful shutdown, let’s take a look at what an ungraceful stop of a service looks like.

When we need to restart a service, we first need to stop it. There are two ways to stop our service:

  • Typing Ctrl + C in the Linux terminal (which actually sends the SIGINT signal).
  • Sending the SIGTERM signal, for example, using kill or systemctl stop.

When we stop the service using either of the above methods, two problems arise:

  • Some requests are being processed, and if the server exits directly, it will interrupt the client connection and cause the requests to fail.
  • Our program may need to perform some cleanup work, such as waiting for tasks in the process’s task queue to complete or refusing to accept new messages.

These issues can have an impact on the business, so we need a graceful way to stop our application. In Go development, graceful shutdown is usually achieved by intercepting the SIGINT and SIGTERM signals. When these two signals are received, the application process performs some cleanup work, then exits the blocking state and continues executing the remaining code, finally naturally exiting the process.

Let’s start with a simple example of graceful shutdown:

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"time"

	"github.com/gin-gonic/gin"
)

func main() {
	router := gin.Default()
	router.GET("/", func(c *gin.Context) {
		time.Sleep(5 * time.Second)
		c.String(http.StatusOK, "Welcome Gin Server")
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: router,
	}

	go func() {
		// Start the server in a goroutine
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatalf("listen: %s\n", err)
		}
	}()

	quit := make(chan os.Signal)
	signal.Notify(quit, os.Interrupt)
	<-quit // Block and wait to receive data from the channel
	log.Println("Shutdown Server ...")

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) // 5 seconds buffer time to handle existing requests
	defer cancel()
	if err := srv.Shutdown(ctx); err != nil { // Call the graceful shutdown function provided by the net/http package: Shutdown
		log.Fatal("Server Shutdown:", err)
	}
	log.Println("Server exiting")
}

The above code implements the following steps for graceful shutdown:

  1. Run the HTTP server in a goroutine, so the program does not block and continues execution.
  2. Create an unbuffered channel quit and use signal.Notify(quit, os.Interrupt) to send the os.Interrupt (SIGINT) signal received by the process to the quit channel.
  3. <-quit blocks the current goroutine (the one in the main function) and waits to receive the shutdown signal from the quit channel. By performing the above steps, we successfully start the HTTP server and block the main function to prevent the goroutine running the HTTP server from exiting. When we type Ctrl + C, the process receives the SIGINT signal and sends it to the quit channel. At this point, <-quit receives the data sent from the other end of the channel, exits the blocking state, and the program continues execution. Here, the purpose of <-quit is to block the current goroutine, so the received data is directly discarded.
  4. Print the shutdown message to indicate that the current service is ready to exit.
  5. Call the Shutdown method provided by the net/http package. The Shutdown method will handle existing requests within the specified time and return.
  6. Finally, after the program executes the code log.Println("Server exiting"), it exits the main function.

iam-apiserver also implements graceful shutdown, and the idea of graceful shutdown is similar to the code above. Specifically, it can be divided into three steps, as follows:

Step 1: Create a channel to receive os.Interrupt (SIGINT) and syscall.SIGTERM (SIGKILL) signals.

The code can be found in internal/pkg/server/signal.go.

var onlyOneSignalHandler = make(chan struct{})

var shutdownHandler chan os.Signal

func SetupSignalHandler() <-chan struct{} {
	close(onlyOneSignalHandler) // panics when called twice

	shutdownHandler = make(chan os.Signal, 2)

	stop := make(chan struct{})

	signal.Notify(shutdownHandler, shutdownSignals...)

	go func() {
		<-shutdownHandler
		close(stop)
		<-shutdownHandler
		os.Exit(1) // second signal. Exit directly.
	}()

	return stop
}

In the SetupSignalHandler function, close(onlyOneSignalHandler) is used to ensure that the code of the iam-apiserver component only calls the SetupSignalHandler function once. Otherwise, it may cause signal loss due to the signals being sent to different shutdownHandlers.

The SetupSignalHandler function also implements a feature: graceful shutdown when receiving one SIGINT/SIGTERM signal, forceful shutdown when receiving two SIGINT/SIGTERM signals. The implementation code is as follows:

go func() {
	<-shutdownHandler
	close(stop)
	<-shutdownHandler
	os.Exit(1) // second signal. Exit directly.
}()

Note: The signal.Notify(c chan<- os.Signal, sig ...os.Signal) function does not block to send information to c. In other words, if c is blocked when sending, the signal package will directly drop the signal. To avoid signal loss, we create a buffered channel shutdownHandler.

Finally, the SetupSignalHandler function returns stop so that the following code can end the blocking state by closing stop.

Step 2: Pass the stop channel to the function that starts the HTTP(S) and gRPC services, start the HTTP(S) and gRPC services as goroutines in the function, and then execute <-stop to block the goroutine.

Step 3: When the iam-apiserver process receives the SIGINT/SIGTERM signal, close the stop channel, and continue executing the code after <-stop. In the subsequent code, we can perform some cleanup logic or call the graceful shutdown functions GracefulStop and Shutdown provided by the google.golang.org/grpc and net/http packages. For example, the following code (located in the internal/apiserver/grpc.go file):

func (s *grpcAPIServer) Run(stopCh <-chan struct{}) {
	listen, err := net.Listen("tcp", s.address)
	if err != nil {
		log.Fatalf("failed to listen: %s", err.Error())
	}

	log.Infof("Start grpc server at %s", s.address)

    go func() {
        if err := s.Serve(listen); err != nil {
            log.Fatalf("failed to start grpc server: %s", err.Error())
        }
    }()

    <-stopCh

    log.Infof("Grpc server on %s stopped", s.address)
    s.GracefulStop()
}

In addition to the above method, the iam-apiserver also implements another graceful shutdown method through the github.com/marmotedu/iam/pkg/shutdown package. This method is more friendly and flexible. The implementation code can be seen in the PrepareRun function.

The usage of the github.com/marmotedu/iam/pkg/shutdown package is as follows:

package main

import (
    "fmt"
    "time"
    "github.com/marmotedu/iam/pkg/shutdown"
    "github.com/marmotedu/iam/pkg/shutdown/shutdownmanagers/posixsignal"
)

func main() {
    // initialize shutdown
    gs := shutdown.New()
    // add posix shutdown manager
    gs.AddShutdownManager(posixsignal.NewPosixSignalManager())
    // add your tasks that implement ShutdownCallback
    gs.AddShutdownCallback(shutdown.ShutdownFunc(func(string) error {
        fmt.Println("Shutdown callback start")
        time.Sleep(time.Second)
        fmt.Println("Shutdown callback finished")
        return nil
    }))
    // start shutdown managers
    if err := gs.Start(); err != nil {
        fmt.Println("Start:", err)
        return
    }
    // do other stuff
    time.Sleep(time.Hour)
}

In the above code, a shutdown instance is created with gs := shutdown.New(). The AddShutdownManager method is used to add a listening signal. The AddShutdownCallback method is used to set the callback function to be executed when the specified signal is received. These callback functions can perform some cleanup work. Finally, the shutdown instance is started with the Start method.

Health Checks #

Usually, we determine the health of the iam-apiserver based on whether the process exists, for example, by executing ps -ef|grep iam-apiserver. However, in actual development, I found that sometimes the service process still exists, but the HTTP service cannot receive or process requests. So a more reliable method is to directly request the health check interface of the iam-apiserver.

After starting the iam-apiserver process, we can manually call the health check interface of the iam-apiserver for checking. However, there is a more convenient way: automatically call the health check interface after starting the service. You can see the implementation of this method in the ping method provided by the GenericAPIServer. In the ping method, pay attention to the following code:

url := fmt.Sprintf("http://%s/healthz", s.InsecureServingInfo.Address)
if strings.Contains(s.InsecureServingInfo.Address, "0.0.0.0") {
    url = fmt.Sprintf("http://127.0.0.1:%s/healthz", strings.Split(s.InsecureServingInfo.Address, ":")[1])
}

When the HTTP service is listening on all interfaces, the requested IP is 127.0.0.1. When the HTTP service is listening on a specified interface, we need to request the IP address of that interface.

Pluginized Loading of Middleware #

iam-apiserver supports the pluginized loading of Gin middleware. With this plugin mechanism, we can choose middleware based on our needs.

Why make middleware a pluginized mechanism? On the one hand, each middleware completes a certain functionality, which is not always needed in all situations. On the other hand, middleware is an additional processing function appended to the HTTP request chain, which may affect the performance of API interfaces. To ensure the performance of API interfaces, we also need to selectively load middleware.

For example, in a test environment, for easy debugging, we can choose to load the dump middleware. The dump middleware can print request and response information, which can assist us in debugging. However, in a production environment, we don’t need the dump middleware for debugging assistance, and if it is loaded, a large amount of request information will be printed during requests, seriously affecting the performance of API interfaces. At this time, we hope that middleware can be loaded on demand.

iam-apiserver uses the InstallMiddlewares function to install Gin middleware. The function code is as follows:

func (s *GenericAPIServer) InstallMiddlewares() {
    // necessary middlewares
    s.Use(middleware.RequestID())
    s.Use(middleware.Context())

    // install custom middlewares
    for _, m := range s.middlewares {
        mw, ok := middleware.Middlewares[m]
        if !ok {
            log.Warnf("cannot find middleware: %s", m)

            continue
        }

        log.Infof("install middleware: %s", m)
        s.Use(mw)
    }
}

As you can see, when installing middleware, we not only install some necessary middleware, but also install some configurable middleware.

The above code installs two default middleware: RequestID and Context.

The RequestID middleware is mainly used to set the X-Request-ID header in the HTTP request and response headers. If the X-Request-ID HTTP header is not present in the HTTP request header, a 64-bit UUID is created. If it exists, it is reused. The UUID is generated using the NewV4().String() method provided by the github.com/satori/go.uuid package:

rid = uuid.NewV4().String()

In addition, there is a design specification for Go constants that you need to pay attention to: constants should be placed together with the package related to that constant, and constants for a project should not be centralized in a package like const. For example, in the requestid.go file, we define the XRequestIDKey = "X-Request-ID" constant. If other places need to use XRequestIDKey, they only need to import the package where XRequestIDKey is located and use it.

The Context middleware is used to set the requestID and username keys in the gin.Context. When logging, the gin.Context variable is passed to the log.L() function, and the log.L() function will output the requestID and username fields in the log output:

2021-07-09 13:33:21.362 DEBUG   apiserver       v1/user.go:106  get 2 users from backend storage.       {"requestID": "f8477cf5-4592-4e47-bdcf-82f7bde2e2d0", "username": "admin"}

The requestID and username fields can facilitate filtering and viewing logs in the future.

In addition to the default middleware, iam-apiserver also supports some configurable middleware. We can configure these middleware by configuring the server.middlewares option in the iam-apiserver configuration file: server.middlewares.

The following middleware can be configured:

  • recovery: Captures any panic and recovers.
  • secure: Adds some security and resource access related HTTP headers.
  • nocache: Disables caching of the HTTP request response.
  • cors: HTTP request cross-origin middleware.
  • dump: Prints the content of the HTTP request and response packages for debugging. Note that loading this middleware is prohibited in production environments.

Of course, you can also add more middleware as needed. The method is simple, just write the middleware and add it to a variable of type map[string]gin.HandlerFunc:

func defaultMiddlewares() map[string]gin.HandlerFunc {
    return map[string]gin.HandlerFunc{
        "recovery":  gin.Recovery(),
        "secure":    Secure,
        "options":   Options,
        "nocache":   NoCache,
        "cors":      Cors(),
        "requestid": RequestID(),
        "logger":    Logger(),
        "dump":      gindump.Dump(),
    }
}

The above code is located in the internal/pkg/middleware/middleware.go file.

There are four features related to coding conventions: API versioning, unified resource metadata, unified response, and concurrent processing template.

API Versioning #

In order to facilitate future expansion, RESTful APIs need to support API versioning. In the 12th lecture, we introduced three methods of API versioning. iam-apiserver chose to include the API version number in the URL, for example, /v1/secrets. The advantage of placing it in the URL is that it is intuitive and easy to see the version number from the API path. Additionally, the API path can also be well mapped to the code paths of the control layer, business layer, and model layer. For example, the storage location for code related to the secret resource is as follows:

internal/apiserver/controller/v1/secret/  # storage location of control layer code
internal/apiserver/service/v1/secret.go # storage location of business layer code
github.com/marmotedu/api/apiserver/v1/secret.go # storage location of model layer code

Regarding the storage path of the code, I have some more points to share with you. For the Secret resource, we usually need to provide CRUD interfaces.

  • C: Create (create Secret).
  • R: Get (get details), List (get list of Secret resources).
  • U: Update (update Secret).
  • D: Delete (delete specified Secret), DeleteCollection (batch delete Secret).

Each interface is independent. In order to reduce the situation where updating code of interface A affects code of interface B due to accidental operation, it is recommended to have one file for each CRUD interface to physically isolate the code of different interfaces. This type of interface also facilitates finding the location of interface A’s code. For example, the storage method for Secret control layer-related code is as follows:

$ ls internal/apiserver/controller/v1/secret/
create.go  delete_collection.go  delete.go  doc.go  get.go  list.go  secret.go  update.go

The same organization can be applied to the business layer and model layer code. In iam-apiserver, because there is not much business logic or model layer code for the Secret, I put it in the internal/apiserver/service/v1/secret.go and github.com/marmotedu/api/apiserver/v1/secret.go files. If the amount of Secret business code increases later, we can also modify it to the following format:

$ ls internal/apiserver/service/v1/secret/
create.go  delete_collection.go  delete.go  doc.go  get.go  list.go  secret.go  update.go

Here’s a side note: both /v1/secret/ and /secret/v1/ directory organization methods are acceptable. You can choose the one you like.

When we need to upgrade the API version, the relevant code can be directly placed in the v2 directory, for example:

internal/apiserver/controller/v2/secret/ # storage location of v2 version control layer code
internal/apiserver/service/v2/secret.go # storage location of v2 version business layer code
github.com/marmotedu/api/apiserver/v2/secret.go # storage location of v2 version model layer code

This way, v1 version code can be physically isolated from v2 version code, they do not interfere with each other, and it is easy to find v2 version code.

Unified Resource Metadata #

One of the highlights of iam-apiserver’s design is that it supports unified resource metadata, just like Kubernetes REST resources.

In iam-apiserver, all resources are REST resources, and iam-apiserver further standardizes the attributes of REST resources. Here, standardization refers to the fact that all REST resources support two types of attributes:

  • Common attributes.
  • Resource-specific attributes.

For example, the definition of the Secret resource is as follows:

type Secret struct {
    // May add TypeMeta in the future.
    // metav1.TypeMeta `json:",inline"`

    // Standard object's metadata.
    metav1.ObjectMeta `       json:"metadata,omitempty"`
    Username          string `json:"username"           gorm:"column:username"  validate:"omitempty"`
    SecretID          string `json:"secretID"           gorm:"column:secretID"  validate:"omitempty"`
    SecretKey         string `json:"secretKey"          gorm:"column:secretKey" validate:"omitempty"`

    // Required: true
    Expires     int64  `json:"expires"     gorm:"column:expires"     validate:"omitempty"`
    Description string `json:"description" gorm:"column:description" validate:"description"`
}

The resource-specific attributes may vary depending on the resource. Let’s take a closer look at the common attribute ObjectMeta. Its definition is as follows:

type ObjectMeta struct {
    ID uint64 `json:"id,omitempty" gorm:"primary_key;AUTO_INCREMENT;column:id"`
    InstanceID string `json:"instanceID,omitempty" gorm:"unique;column:instanceID;type:varchar(32);not null"`
    Name string `json:"name,omitempty" gorm:"column:name;type:varchar(64);not null" validate:"name"`
    Extend Extend `json:"extend,omitempty" gorm:"-" validate:"omitempty"`
    ExtendShadow string `json:"-" gorm:"column:extendShadow" validate:"omitempty"`
    CreatedAt time.Time `json:"createdAt,omitempty" gorm:"column:createdAt"`

4.    UpdatedAt time.Time `json:"updatedAt,omitempty" gorm:"column:updatedAt"`
}

Next, I will explain the meaning and purpose of each field in detail in the public property.

  1. ID

The ID here is mapped to the id field in the MariaDB database. In some applications, the id field is used as the unique identifier of a resource. However, in iam-apiserver, ID is not used as the unique identifier of a resource, but InstanceID is used instead. The ID in iam-apiserver is only used to map with the database id field, and it is not used in the code.

  1. InstanceID

InstanceID is the unique identifier of a resource, and it has the format <resource identifier>-xxxxxx. In this format, <resource identifier> is the English identifier of the resource, and xxxxxx is a random string. The character set is abcdefghijklmnopqrstuvwxyz1234567890, and the length is >=6, for example, secret-yj8m30, user-j4lz3g, policy-3v18jq.

Tencent Cloud, Alibaba Cloud, and Huawei Cloud also use this format of string as the unique identifier of resources.

The generation and updating of InstanceID are automated. After the record is inserted into the secret table of the iam database, the AfterCreate Hooks provided by gorm are used to generate and update the instanceID field in the database:

func (s *Secret) AfterCreate(tx *gorm.DB) (err error) {
    s.InstanceID = idutil.GetInstanceID(s.ID, "secret-")

    return tx.Save(s).Error
}

In the above code, after the Secret record is inserted into the secret table of the iam database, the idutil.GetInstanceID is called to generate the InstanceID, and tx.Save(s) is used to update the instanceID field in the secret table of the database.

Since in most cases, REST resources in an application are stored in a single table in the database, this ensures that the database ID of each resource in the application is unique. Therefore, the GetInstanceID(uid uint64, prefix string) string function uses the method provided by the github.com/speps/go-hashids package to hash the database ID into a unique string at the database level (e.g., 3v18jq), and this string is used as the InstanceID of the resource based on the provided prefix.

Using this approach to generate the unique identifier of a resource has the following advantages:

  • Unique at the database level.
  • InstanceID is a string with a controllable length. The minimum length is 6 characters, but it can dynamically increase in length based on the number of records in the table. Based on my tests, InstanceID generated within 2176782336 records will have a length of no more than 6 characters. Another advantage of a controllable length is easy memorization and propagation.

Here, please note that if the same resource is stored in different tables, the generated InstanceIDs may be the same. However, the probability of this happening is extremely low, almost zero. In such cases, we need to use distributed ID generation techniques. However, this is another topic, which will not be further explained here.

In practical development, many developers use the database incrementing ID field (e.g., 121) or 36/64-bit UUID (e.g., 20cd59d4-08c6-4e86-a9d4-a0e51c420a04) as the unique identifier of a resource. Compared to these two resource identifier methods, using <resource identifier>-xxxxxx has the following advantages:

  • By looking at the identifier, we can easily determine the type of resource it represents. For example, secret-yj8m30 indicates that it is a resource of the secret type. This can effectively reduce operational errors during troubleshooting.
  • The length is controllable, and it occupies a small amount of database space. The resource identifier in iam-apiserver can be considered to have a length of 12 characters at most (6 characters for secret/policy, plus 6 random characters).
  • If numeric values like 121 are used as resource unique identifiers, it indirectly reveals the scale of the system to competitors, which must be prohibited.

Additionally, there are some systems, such as Kubernetes, that use resource names as unique identifiers. This approach has a disadvantage that when there are too many resources of the same type in the system, creating resources easily leads to naming conflicts, making it difficult for you to use the name you want. Therefore, iam-apiserver does not adopt this design approach.

We use instanceID as the unique identifier of a resource, and in the code, we often need to query resources based on their instanceID. Therefore, in the database, this field should be set as a unique index, which can prevent duplicate instanceIDs and improve query speed.

  1. Name

Name is the name of the resource, and we can easily identify a resource based on its name.

  1. Extend, ExtendShadow

Extend and ExtendShadow are another highlight of iam-apiserver’s design.

In practical development, we often encounter this problem: as the business evolves, a resource may need to add some properties. In this case, we may choose to add a new database field. However, as the business system evolves, the number of fields in the database keeps increasing, and our code needs to adapt accordingly, which ultimately becomes difficult to maintain.

We may also encounter a situation where we store the above-mentioned fields in the meta field in the database, and the data format of the meta field in the database is {"disable":true,"tag":"colin"}. However, if we want to use these fields in the code, we need to unmarshal the data into a struct, for example:

metaData := `{"disable":true,"tag":"colin"}`
meta := make(map[string]interface{})
if err := json.Unmarshal([]byte(metaData), &meta); err != nil {
    return err
}

When storing it back into the database, we need to marshal it into a JSON-formatted string, for example:

meta := map[string]interface{}{"disable": true, "tag": "colin"}
data, err := json.Marshal(meta)
if err != nil {
    return err
}

You can see that these Unmarshal and Marshal operations are a bit cumbersome.

Because each resource may need to use extended fields, is there a universal solution? iam-apiserver solves this problem through Extend and ExtendShadow.

Extend is a field of type Extend, which is actually an alias for map[string]interface{}. In the program, we can conveniently refer to the attributes contained in Extend, which are the keys of the map. When the Extend field is saved in the database, it will be automatically Marshaled into a string and saved in the ExtendShadow field.

ExtendShadow is the shadow of Extend in the database. Similarly, when querying data from the database, the value of ExtendShadow will be automatically Unmarshaled into a variable of type Extend for program usage.

The implementation is as follows:

  • With the help of BeforeCreate and BeforeUpdate Hooks provided by gorm, the value of Extend is converted to a string and saved in the ExtendShadow field when inserting or updating records. Finally, it is saved in the ExtendShadow field of the database.
  • With the help of the AfterFind Hook provided by gorm, the value of ExtendShadow is Unmarshaled into the Extend field after querying data, and then the program can use the attributes in the Extend field.

  1. CreatedAt

    The creation time of the resource. We should record the creation time of each resource when it is created, which can help with troubleshooting, analysis, etc.

  2. UpdatedAt

    The update time of the resource. We should record the update time of each resource when it is updated. This field is automatically updated by gorm when the resource is updated.

As you can see, the ObjectMeta structure contains many fields, each of which has completed cool functionality. So if ObjectMeta is used as a common attribute for all resources, these resources will have these capabilities.

Of course, some developers may say that the User resource does not really need the resource identifier like user-xxxxxx, so the InstanceID field is actually a useless field. But in my opinion, compared with redundant functionality, standardized functionality, avoiding reinventing the wheel, and the other functionalities of ObjectMeta are more important. Therefore, it is also recommended to use unified resource metadata for all REST resources.

Unified Return #

In Chapter 18, we introduced that the interface return format of the API should be uniform. The best way to return a fixed format message is to use the same return function. Since API interfaces are all returned through the same function, the return format is naturally unified.

The IAM project uses the WriteResponse function provided by the github.com/marmotedu/component-base/pkg/core package to return the result. The definition of the WriteResponse function is as follows:

    func WriteResponse(c *gin.Context, err error, data interface{}) {
        if err != nil {
            log.Errorf("%#+v", err)
            coder := errors.ParseCoder(err)
            c.JSON(coder.HTTPStatus(), ErrResponse{
                Code:      coder.Code(),
                Message:   coder.String(),
                Reference: coder.Reference(),
            })
    
            return
        }
    
        c.JSON(http.StatusOK, data)
    }

As you can see, the WriteResponse function checks whether the err is nil. If it is not nil, it parses err into an error of type Coder defined in the github.com/marmotedu/errors package, and calls the Code(), String(), and Reference() methods provided by the Coder interface to obtain the business code, the error message shown to the outside world, and the troubleshooting document of the error. If err is nil, the function calls c.JSON to return the data in JSON format.

Concurrency Processing Template #

In Go project development, we often encounter a scenario: when querying a list interface, multiple records are returned, but some other logical processing needs to be done for each record. Because there are multiple records, such as 100 records, if the time delay for processing each record is X milliseconds, the total delay for processing all 100 records serially would be 100 * X milliseconds. If X is relatively large, the overall delay for complete processing would be very high, which would seriously affect the performance of the API interface.

At this time, we naturally think of using the multi-core capability of the CPU for concurrent processing of these 100 records. We often encounter this kind of scenario in actual development, so it is necessary to abstract it into a concurrency processing template so that it can be used later when querying.

For example, in iam-apiserver, when querying the user list interface List, the number of policies owned by each user also needs to be returned. This uses concurrency processing. Here, I tried to abstract it into a template, and the template is as follows:

    func (u *userService) List(ctx context.Context, opts metav1.ListOptions) (*v1.UserList, error) {
        users, err := u.store.Users().List(ctx, opts)
        if err != nil {
            log.L(ctx).Errorf("list users from storage failed: %s", err.Error())

            return nil, errors.WithCode(code.ErrDatabase, err.Error())
        }

        wg := sync.WaitGroup{}
        errChan := make(chan error, 1)
        finished := make(chan bool, 1)

        var m sync.Map

        // Improve query efficiency in parallel
        for _, user := range users.Items {
            wg.Add(1)

            go func(user *v1.User) {
                defer wg.Done()

                // some cost time process
                policies, err := u.store.Policies().List(ctx, user.Name, metav1.ListOptions{})
                if err != nil {

errChan <- errors.WithCode(code.ErrDatabase, err.Error())

return
}

m.Store(user.ID, &v1.User{
    ...
    Phone:       user.Phone,
    TotalPolicy: policies.TotalCount,
})

In the above concurrent template, I have implemented three functionalities to process the query results concurrently:

The first functionality is to return if goroutine encounters an error. When the code segment in a goroutine encounters an error, the error message will be written into errChan. In the List function, we use the select statement to return as soon as an error occurs:

select {
case <-finished:
case err := <-errChan:
    return nil, err
}

The second functionality is to maintain the order of queries. The list returned from the database is ordered, such as by default ascending order based on the database ID field, or any other specified sorting method. This order gets disrupted during concurrent processing. However, to ensure that the final returned results are sorted as expected, we need to maintain the same order as the query results in the concurrent template.

In the template above, we save the processed records in a map, where the map’s key is the database ID. And then, we retrieve the records from the map based on the ID in the order of the query. For example:

var m sync.Map
for _, user := range users.Items {
    ...
    go func(user *v1.User) {
        ...
        m.Store(user.ID, &v1.User{})
    }(user)
}
...
infos := make([]*v1.User, 0, len(users.Items))
for _, user := range users.Items {
    info, _ := m.Load(user.ID)
    infos = append(infos, info.(*v1.User))
}

By using this approach, we can ensure that the final returned results follow the same order as the results obtained from the database query.

The third functionality is concurrency safety. In Go language, the map is not concurrency-safe. To achieve concurrency safety, we need to implement it ourselves (such as using locks), or use sync.Map. The template above uses sync.Map.

Of course, if we expect the List interface to return within a certain time, we can also add a timeout mechanism, for example:

select {
case <-finished:
case err := <-errChan:
    return nil, err
case <-time.After(time.Duration(30 * time.Second)):
    return nil, fmt.Errorf("list users timeout after 30 seconds")

}

Although goroutines are lightweight, they still consume resources. If we need to handle hundreds or thousands of concurrency, we can use a coroutine pool to reuse coroutines and save resources. There are many excellent coroutine packages available for us to directly use, such as ants, tunny, etc.

Other Features #

In addition to the two categories mentioned above, I would also like to introduce you to other features in key code design, including plugin-based JSON library selection, call chaining implementation, and data consistency.

Plugin-based JSON Library Selection #

The standard JSON parsing library encoding/json provided by Golang has performance issues when developing high-performance and high-concurrency network services. Therefore, many developers often choose third-party high-performance JSON parsing libraries in actual development, such as jsoniter, easyjson, and jsonparser.

I have seen many developers choose jsoniter, and some developers have used easyjson. jsoniter has slightly better performance than encoding/json. However, with the iteration of Go versions, the performance of encoding/json library has also improved, and the performance advantage of jsoniter has become more limited. Therefore, the IAM project uses the jsoniter library and is prepared to switch back to the encoding/json library at any time.

To facilitate the switch between different JSON packages, the iam-apiserver uses a plugin-based mechanism to use different JSON packages. Specifically, it is implemented by using Go’s build tags to dynamically select the parsing library to be used at runtime.

Build tags are annotations added in source code files near the top of the file using comments. When go build is used to build a package, it reads each source file in the package and analyzes the build tags, which determine whether the source file is included in the current build. For example:

// +build jsoniter

package json

import jsoniter "github.com/json-iterator/go"

Here, +build jsoniter is the build tag. It is important to note that a source file can have multiple build tags, and these tags are evaluated as a logical “AND” relationship. A build tag can also include multiple tags separated by spaces, which means they have a logical “OR” relationship. For example:

// +build linux darwin
// +build 386

It is important to have a blank line separating the build tags and the package declaration; otherwise, the build tags will be treated as comments for the package declaration, rather than build tags.

Specifically, how do we achieve plugin-based JSON library selection?

First, I defined a custom json package at github.com/marmotedu/component-base/pkg/json to adapt to both encoding/json and json-iterator. The github.com/marmotedu/component-base/pkg/json package consists of two files:

  • json.go: Maps the Marshal, Unmarshal, MarshalIndent, NewDecoder, and NewEncoder methods of the encoding/json package.
  • jsoniter.go: Maps the Marshal, Unmarshal, MarshalIndent, NewDecoder, and NewEncoder methods of the github.com/json-iterator/go package.

By using build tags, json.go and jsoniter.go allow the Go compiler to select which file to use during the build process.

Then, by specifying the -tags parameter when executing go build, we can choose which JSON file to compile. For example, json/json.go or json/jsoniter.go.

At the top of both json/json.go and json/jsoniter.go Go files, there is a comment line:

// +build !jsoniter

// +build jsoniter

These comments are used as build tags to specify different conditions for Go compilation. // +build !jsoniter means that this Go file will be compiled when the tags are not “jsoniter”. // +build jsoniter means that this Go file will be compiled when the tags are “jsoniter”. In other words, these two conditions are mutually exclusive. Only when the tags are “jsoniter” will json-iterator be used, otherwise encoding/json will be used.

For example, if we want to use the package, we can compile the project like this:

$ go build -tags=jsoniter

In actual development, we need to choose the appropriate JSON library based on the scenario. Here are some suggestions.

Scenario 1: Struct serialization and deserialization

In this scenario, I personally recommend the official JSON library. You may be surprised, so let me explain my reasons:

First, although easyjson’s performance surpasses all other open-source projects, it has a major drawback, which is the need to use additional tools to generate the code, thereby increasing operational and maintenance costs. Of course, if your team is already proficient in handling protobuf, you can manage easyjson in the same way.

Second, although the official JSON library’s performance was often criticized before Go 1.8, it is now (1.16.3) significantly improved. In addition, as the most widely used JSON library with the least bugs and the best compatibility, the official library is a good choice.

Lastly, while jsoniter still has better performance than the official library, it doesn’t reach an extreme level. If you pursue extreme performance, you should choose easyjson instead of jsoniter. jsoniter has been inactive in recent years. For example, I submitted an issue some time ago and received no response. So I went to check the issue list and found that there were still some issues from 2018.

Scenario 2: Serialization and deserialization of non-structured data

In this scenario, we should consider two cases: high data utilization and low data utilization. If you’re not familiar with data utilization, let me give you an example: if more than a quarter of the data in the JSON body is what the business needs to focus on and process, it can be considered as high data utilization.

In the case of high data utilization, I recommend using jsonvalue.

As for low data utilization, it can be further divided into two cases based on whether the JSON data needs to be reserialized.

If no reserialization is needed, you can simply use jsonparser, because its performance is outstanding.

If reserialization is needed, you have two choices: if performance requirements are relatively low, you can use jsonvalue; if high performance is required and you only need to insert one piece of data into the binary sequence, you can use the Set method of jsoniter.

In actual operations, there are very few cases where we have both a large amount of JSON data and the need for reserialization. These cases often occur in proxy servers, gateways, overlay relay services, etc., when we also need to inject additional information into the original data. In other words, jsoniter’s applicable scenarios are relatively limited.

The following chart shows the efficiency comparison of different libraries for data coverage rates ranging from 10% to 60% (vertical axis unit: μs/op):

Image

As we can see, when the data utilization of jsoniter reaches 25%, it no longer has any advantage compared to jsonvalue and jsonparser. As for jsonvalue, since it performs full parsing of the data once, the access time after parsing is very minimal, so the time consumption remains stable for different data coverage rates.

Call Chain Implementation #

Call chains are very useful for log lookup and troubleshooting. Therefore, the iam-apiserver also implements call chains by using the requestID to connect the entire call chain.

This is done through the following two steps:

  1. Pass the variable ctx context.Context as the first parameter when calling a function.

  2. Use log.L(ctx context.Context) in different functions to log the information.

When a request arrives, the request will be processed by the Context middleware:

func Context() gin.HandlerFunc {
	return func(c *gin.Context) {
		// Sets the `log.KeyRequestID` key in the variable of type `gin.Context`, with a value of a 36-character UUID
		c.Set(log.KeyRequestID, c.GetString(XRequestIDKey))
		// Sets the `log.KeyUsername` key in the variable of type `gin.Context`, with a value of a username
		c.Set(log.KeyUsername, c.GetString(UsernameKey))
		// Calls the next middleware/handler in the chain
		c.Next()
	}
}

In the Context middleware, the log.KeyRequestID key is set in the variable of type gin.Context with a value of a 36-character UUID. The UUID is generated by the RequestID middleware and set in the context of the gin request.

The RequestID middleware is loaded before the Context middleware, so when the Context middleware is executed, it can retrieve the UUID generated by the RequestID.

The log.L(ctx context.Context) function retrieves the log.KeyRequestID from the context ctx to include it as an additional field when logging.

With the above approach, we can establish the request call chain in the iam-apiserver, and the logging example is as follows:

2021-07-19 19:41:33.472 INFO    apiserver       apiserver/auth.go:205   user `admin` is authenticated.  {"requestID": "b6c56cd3-d095-4fd5-a928-291a2e33077f", "username": "admin"}
2021-07-19 19:41:33.472 INFO    apiserver       policy/create.go:22     create policy function called.  {"requestID": "b6c56cd3-d095-4fd5-a928-291a2e33077f", "username": "admin"}
...

In addition, having the ctx context.Context as the first parameter of a function/method also has the advantage of facilitating future extension. For example, if we have the following call relationship:

package main

import "fmt"

func B(name, address string) string {
    return fmt.Sprintf("name: %s, address: %s", name, address)
}

func A() string {
    return B("colin", "sz")
}

func main() {
    fmt.Println(A())
}

The above code ultimately calls the B function to print the username and address. If, as the business evolves, we want to pass the phone number of the user when calling A, and have B print the user’s phone number, we may consider adding a phone number parameter to the B function, for example:

func B(name, address, phone string) string {
    return fmt.Sprintf("name: %s, address: %s, phone: %s", name, address, phone)
}

But what if we also want to add age, gender, and other attributes later? Continuously adding parameters to the B function in this way is not only troublesome, but also requires modifying all functions that call B, which can be a lot of work. In this case, we can consider passing these additional parameters through the ctx context.Context to achieve the following:

package main

import (
    "context"
    "fmt"
)

func B(ctx context.Context, name, address string) string {
    return fmt.Sprintf("name: %s, address: %s, phone: %v", name, address, ctx.Value("phone"))
}

func A() string {
    ctx := context.WithValue(context.TODO(), "phone", "1812884xxxx")
    return B(ctx, "colin", "sz")
}

func main() {
    fmt.Println(A())
}

This way, the next time we need to add a new parameter, we only need to call the context’s WithValue method:

ctx = context.WithValue(ctx, "sex", "male")

In the B function, we can then retrieve the sex key from the context through the Value method provided by the context.Context variable:

return fmt.Sprintf("name: %s, address: %s, phone: %v, sex: %v", name, address, ctx.Value("phone"), ctx.Value("sex"))

Data Consistency #

In order to improve the responsiveness of the iam-authz-server, I cache the secret keys and authorization policy information in the memory of the iam-authz-server deployment machine. At the same time, for high availability, we need to ensure that the number of instances of iam-authz-server started is at least two. In this case, we will face the issue of data consistency: all the data cached by iam-authz-server needs to be consistent and consistent with what is saved in the iam-apiserver database. iam-apiserver achieves data consistency through the following steps:

The specific process is as follows:

  • Step 1: When iam-authz-server starts, it uses grpc to call the GetSecrets and GetPolicies interfaces of iam-apiserver to obtain all secret keys and authorization policy information.

  • Step 2: When we invoke the write interfaces (POST, PUT, DELETE) for secret keys/authorization policies of iam-apiserver through the console, a SecretChanged/PolicyChanged message is sent to the iam.cluster.notifications channel in Redis.

  • Step 3: iam-authz-server subscribes to the iam.cluster.notifications channel. When a SecretChanged/PolicyChanged message is received, it requests iam-apiserver to fetch all secret keys/authorization policies.

By using the Redis Sub/Pub mechanism, it ensures that the cached data of each iam-authz-server node is consistent with the data saved in the iam-apiserver database. All nodes fetch data by calling the same interface of iam-apiserver, thereby ensuring that the data of all iam-authz-server nodes is consistent.

Summary #

Today, I have shared with you some key features and implementation of iam-apiserver, as well as introduced my design thoughts. Here, let me summarize briefly:

  • In order to ensure that HTTP requests can complete execution before disconnecting when the process is shut down, iam-apiserver implements the graceful shutdown feature.
  • To avoid the scenario where the process exists but the service fails to start, iam-apiserver implements a health check mechanism.
  • Gin middleware can be configured through the configuration file, thus achieving the feature of on-demand loading.
  • In order to directly identify the version of the API, iam-apiserver puts the version identifier of the API in the URL path, for example, /v1/secrets.
  • In order to maximize the sharing of functional code, iam-apiserver abstracts a unified metadata that each REST resource has.
  • Because the API interfaces are all returned through the same function, the return format is naturally standardized.
  • Since concurrent logic is often needed in the program, iam-apiserver abstracts a common concurrency template.
  • In order to switch JSON libraries according to needs, we have implemented the functionality of plugin-based JSON library selection.
  • In order to implement the call chain feature, iam-apiserver passes the RequestID between different functions through ctx context.Context.
  • iam-apiserver ensures data consistency through Redis’s Sub/Pub mechanism.

Post-class Exercises #

  1. Think about the better methods of concurrent processing that you have used in your project development. Feel free to share in the comment section.

  2. Try adding a new configurable Gin middleware to iam-apiserver to implement API rate limiting.

Feel free to communicate and discuss with me in the comment section. See you in the next lesson.