28 Control Flow Designing iamapiserver and Looking at the Construction of Web Services #

Hello, I’m Kong Lingfei.

In the previous discussions, we talked a lot about application building. You must be eager to see how the IAM project is built. So next, I will explain the source code of the IAM application.

During the explanation, I won’t go into the details of coding, but I will explain some key points and challenges in the building process, as well as the design thoughts and ideas behind the code. I believe this will be more helpful to you.

The IAM project consists of many components. In this lesson, I will first introduce the facade service of the IAM project: iam-apiserver (the management flow service). I will first introduce the functionality and usage of iam-apiserver, and then explain the code implementation of iam-apiserver.

Introduction to iam-apiserver Service #

iam-apiserver is a web service that provides RESTful API interfaces through a process called iam-apiserver. It supports the CRUD operations for three types of resources: users, keys, and policies. Now, I will introduce the service in terms of its functionalities and usage.

Functionality of iam-apiserver #

We can understand the functionalities provided by iam-apiserver by examining the RESTful API interfaces it offers. The RESTful API interfaces provided by iam-apiserver can be categorized into four types, as follows:

Authentication related interfaces

User related interfaces

Key related interfaces

Policy related interfaces

Usage of iam-apiserver #

After introducing the functionalities of iam-apiserver, let me explain how to use these functionalities.

We can access iam-apiserver through different clients, such as frontend, API calls, SDKs, and iamctl. All these clients eventually execute HTTP requests to call the RESTful API interfaces provided by iam-apiserver. Therefore, we need a handy REST API client tool to execute HTTP requests for development and testing purposes.

Since different developers prefer different ways of executing HTTP requests, here I will use the cURL tool as a common example. Now let me introduce the cURL tool.

The cURL tool is installed in standard Linux distributions. It provides convenient features for making RESTful API calls, such as setting headers, specifying HTTP request methods, assigning HTTP message bodies, providing authentication information, etc. By using the -v option, you can also output all the details of the REST requests. cURL is powerful and has many parameters. Here are some commonly used parameters of the cURL tool:

-X/--request [GET|POST|PUT|DELETE|…]  Specifies the HTTP method of the request
-H/--header                           Specifies the HTTP header of the request
-d/--data                             Specifies the HTTP message body
-v/--verbose                          Outputs detailed return information
-u/--user                             Specifies username and password
-b/--cookie                           Reads the cookie

In addition, if you prefer using a GUI-based tool, I recommend using Insomnia.

Insomnia is a cross-platform REST API client, similar to tools like Postman and Apifox, used for interface management and testing. Insomnia has powerful features, including:

• Sending HTTP requests
• Creating workspaces or folders
• Importing and exporting data
• Exporting HTTP requests in cURL format
• Supporting writing Swagger documents
• Quickly switching requests
• URL encoding and decoding
• …

The interface of Insomnia is shown in the following figure:

Of course, there are many other excellent GUI-based REST API clients, such as Postman and Apifox. You can choose one based on your needs.

Next, I will demonstrate how to use the functionalities of iam-apiserver by performing CRUD operations on the secret resource. You will need to follow six steps:

1. Log in to iam-apiserver and obtain the token.
2. Create a secret named secret0.
3. Retrieve detailed information about secret0.
4. Update the description of secret0.
5. Retrieve the list of secrets.
6. Delete secret0.

Here are the specific operations:

1. Log in to iam-apiserver and obtain the token:

$ curl -s -XPOST -H"Authorization: Basic `echo -n 'admin:Admin@2021'|base64`" http://127.0.0.1:8080/login | jq -r .token
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJpYW0uYXBpLm1hcm1vdGVkdS5jb20iLCJleHAiOjE2MzUwNTk4NDIsImlkZW50aXR5IjoiYWRtaW4iLCJpc3MiOiJpYW0tYXBpc2VydmVyIiwib3JpZ19pYXQiOjE2MjcyODM4NDIsInN1YiI6ImFkbWluIn0.gTS0n-7njLtpCJ7mvSnct2p3TxNTUQaduNXxqqLwGfI

To facilitate usage, we will set the token as an environment variable:

TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJpYW0uYXBpLm1hcm1vdGVkdS5jb20iLCJleHAiOjE2MzUwNTk4NDIsImlkZW50aXR5IjoiYWRtaW4iLCJpc3MiOiJpYW0tYXBpc2VydmVyIiwib3JpZ19pYXQiOjE2MjcyODM4NDIsInN1YiI6ImFkbWluIn0.gTS0n-7njLtpCJ7mvSnct2p3TxNTUQaduNXxqqLwGfI

1. Create a secret named secret0:

$ curl -v -XPOST -H "Content-Type: application/json" -H"Authorization: Bearer ${TOKEN}" -d'{"metadata":{"name":"secret0"},"expires":0,"description":"admin secret"}' http://iam.api.marmotedu.com:8080/v1/secrets
* About to connect() to iam.api.marmotedu.com port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to iam.api.marmotedu.com (127.0.0.1) port 8080 (#0)
> POST /v1/secrets HTTP/1.1
> User-Agent: curl/7.29.0
> Host: iam.api.marmotedu.com:8080
> Accept: */*
> Content-Type: application/json
> Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJpYW0uYXBpLm1hcm1vdGVkdS5jb20iLCJleHAiOjE2MzUwNTk4NDIsImlkZW50aXR5IjoiYWRtaW4iLCJpc3MiOiJpYW0tYXBpc2VydmVyIiwib3JpZ19pYXQiOjE2MjcyODM4NDIsInN1YiI6ImFkbWluIn0.gTS0n-7njLtpCJ7mvSnct2p3TxNTUQaduNXxqqLwGfI
> Content-Length: 72
> 
* upload completely sent off: 72 out of 72 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< X-Request-Id: ff825bea-53de-4020-8e68-4e87574bd1ba
< Date: Mon, 26 Jul 2021 07:20:26 GMT
< Content-Length: 313
< 
* Connection #0 to host iam.api.marmotedu.com left intact
{"metadata":{"id":60,"instanceID":"secret-jedr3e","name":"secret0","createdAt":"2021-07-26T15:20:26.885+08:00","updatedAt":"2021-07-26T15:20:26.907+08:00"},"username":"admin","secretID":"U6CxKs0YVWyOp5GrluychYIRxDmMDFd1mOOD","secretKey":"fubNIn8jLA55ktuuTpXM8Iw5ogdR2mlf","expires":0,"description":"admin secret"}

You can see that the response header contains the X-Request-Id header, which uniquely identifies this request. If this request fails, you can provide the X-Request-Id to operations or developers to locate the failed request for troubleshooting. In addition, in the microservice scenario, X-Request-Id can also be transmitted to other services to implement request call chains.

2. Get detailed information about secret0:

$ curl -XGET -H"Authorization: Bearer ${TOKEN}" http://iam.api.marmotedu.com:8080/v1/secrets/secret0
{"metadata":{"id":60,"instanceID":"secret-jedr3e","name":"secret0","createdAt":"2021-07-26T15:20:26+08:00","updatedAt":"2021-07-26T15:20:26+08:00"},"username":"admin","secretID":"U6CxKs0YVWyOp5GrluychYIRxDmMDFd1mOOD","secretKey":"fubNIn8jLA55ktuuTpXM8Iw5ogdR2mlf","expires":0,"description":"admin secret"}

3. Update the description of secret0:

$ curl -XPUT -H"Authorization: Bearer ${TOKEN}" -d'{"metadata":{"name":"secret"},"expires":0,"description":"admin secret(modify)"}' http://iam.api.marmotedu.com:8080/v1/secrets/secret0
{"metadata":{"id":60,"instanceID":"secret-jedr3e","name":"secret0","createdAt":"2021-07-26T15:20:26+08:00","updatedAt":"2021-07-26T15:23:35.878+08:00"},"username":"admin","secretID":"U6CxKs0YVWyOp5GrluychYIRxDmMDFd1mOOD","secretKey":"fubNIn8jLA55ktuuTpXM8Iw5ogdR2mlf","expires":0,"description":"admin secret(modify)"}

4. Get the list of secrets:

$ curl -XGET -H"Authorization: Bearer ${TOKEN}" http://iam.api.marmotedu.com:8080/v1/secrets
{"totalCount":1,"items":[{"metadata":{"id":60,"instanceID":"secret-jedr3e","name":"secret0","createdAt":"2021-07-26T15:20:26+08:00","updatedAt":"2021-07-26T15:23:35+08:00"},"username":"admin","secretID":"U6CxKs0YVWyOp5GrluychYIRxDmMDFd1mOOD","secretKey":"fubNIn8jLA55ktuuTpXM8Iw5ogdR2mlf","expires":0,"description":"admin secret(modify)"}]}

5. Delete secret0:

$ curl -XDELETE -H"Authorization: Bearer ${TOKEN}" http://iam.api.marmotedu.com:8080/v1/secrets/secret0
null

Above, I have demonstrated the usage of secrets. The usage of user and policy resource types is similar to secrets. For detailed usage, you can refer to the test.sh script, which is used to test the IAM application and contains request methods for each interface.

Here, I would also like to briefly introduce how to test each part of the IAM application. After ensuring that iam-apiserver, iam-authz-server, iam-pump, and other services are running normally, go to the root directory of the IAM project and execute the following commands:

$ ./scripts/install/test.sh iam::test::test # Test if the entire IAM application is running correctly
$ ./scripts/install/test.sh iam::test::login # Test if the login interface can be accessed normally
$ ./scripts/install/test.sh iam::test::user # Test if the user interface can be accessed normally
$ ./scripts/install/test.sh iam::test::secret # Test if the secret interface can be accessed normally
$ ./scripts/install/test.sh iam::test::policy # Test if the policy interface can be accessed normally
$ ./scripts/install/test.sh iam::test::apiserver # Test if the iam-apiserver service is running properly
$ ./scripts/install/test.sh iam::test::authz # Test if the authz interface can be accessed normally
$ ./scripts/install/test.sh iam::test::authzserver # Test if the iam-authz-server service is running properly
$ ./scripts/install/test.sh iam::test::pump # Test if iam-pump is running properly
$ ./scripts/install/test.sh iam::test::iamctl # Test if the iamctl tool can be used correctly
$ ./scripts/install/test.sh iam::test::man # Test if the man file is installed correctly

So, after each release of iam-apiserver, you can execute the following commands to perform smoke tests on iam-apiserver:

$ export IAM_APISERVER_HOST=127.0.0.1 # The IP address of the server where iam-apiserver is deployed
$ export IAM_APISERVER_INSECURE_BIND_PORT=8080 # The listening port of the iam-apiserver HTTP service
$ ./scripts/install/test.sh iam::test::apiserver

Implementation of iam-apiserver code #

Above, I introduced the functionality and usage of iam-apiserver. Here, let’s take a look at the specific implementation of iam-apiserver code. I will explain it from four aspects: configuration processing, startup process, request processing process, and code architecture.

Configuration Processing of iam-apiserver #

The main function of the iam-apiserver service is located in the apiserver.go file. You can read the code to understand the implementation of iam-apiserver. Here, let me introduce some design ideas of the iam-apiserver service.

First, let’s look at the three types of configurations in iam-apiserver: Options configuration, application configuration, and HTTP/GRPC service configuration.

• Options Configuration: It is used to build command-line arguments, and its value comes from command-line options or configuration files (which may be the merged configuration of the two). Options can be used to build the application framework, and the Options configuration is also the input of the application configuration.
• Application Configuration: It includes all the configurations required by the iam-apiserver component. There are many places that need to be configured, such as configuring the listening address and port for starting HTTP/GRPC, and configuring the database address, username, password, etc. for initializing the database.
• HTTP/GRPC Service Configuration: It includes the configurations required to start the HTTP service or GRPC service.

The relationship between these three configurations is shown in the following diagram:

Options configuration handles command-line options, application configuration handles the configuration of the entire application, and HTTP/GRPC service configuration handles the configurations related to HTTP/GRPC services. These three configurations are independent and can decouple command-line options, applications, and services within the application, making these three parts independently extensible without affecting each other.

iam-apiserver builds command-line arguments and application configuration based on the Options configuration.

We use the buildCommand method of the github.com/marmotedu/iam/pkg/app package to build command-line arguments. The core here is that when building the Application instance with the NewApp function, we pass in an instance of Options that implements the Flags() (fss cliflag.NamedFlagSets) method. With the following code in the buildCommand method, the flags of the options are added to the cobra instance’s FlagSet:

    if a.options != nil {
        namedFlagSets = a.options.Flags()
        fs := cmd.Flags()
        for _, f := range namedFlagSets.FlagSets {
            fs.AddFlagSet(f)
        }
        ...
    }

The application configuration is created based on the Options configuration using the CreateConfigFromOptions function:

    cfg, err := config.CreateConfigFromOptions(opts)
    if err != nil {
        return err
    }

Based on the application configuration, the HTTP/GRPC service configuration is created. For example, the following code creates the Address parameter for the HTTP server based on the application configuration:

    func (s *InsecureServingOptions) ApplyTo(c *server.Config) error {
        c.InsecureServing = &server.InsecureServingInfo{
            Address: net.JoinHostPort(s.BindAddress, strconv.Itoa(s.BindPort)),
        }
        return nil
    }

Here, c *server.Config is the configuration of the HTTP server, and s *InsecureServingOptions is the application configuration.

Startup Process Design of iam-apiserver #

Next, let’s take a detailed look at the startup process design of iam-apiserver. The startup process is shown in the following diagram:

First, create an opts variable with default values using opts := options.NewOptions(). The opts variable is used as an input parameter for the NewApp function in the github.com/marmotedu/iam/pkg/app package. Ultimately, in the App framework, it will be filled with the configurations from command-line parameters or configuration files (which may be the merged configuration of the two), and the values of the fields in the opts variable will be used to create the application configuration.

Next, register the run function to the App framework. The run function is the startup function of iam-apiserver, and it encapsulates our custom startup logic. In the run function, the logging package is first initialized, so that we can record logs as needed in the subsequent code.

Then, the application configuration is created. The application configuration and the Options configuration are actually completely independent, and they may be completely different. But in iam-apiserver, the configuration items of the two are the same.

After that, based on the application configuration, the configuration used by the HTTP/GRPC server is created. After creating the configuration, the configurations are completed separately, and then web server instances are created using the completed configurations. For example:

    genericServer, err := genericConfig.Complete().New()
    if err != nil {
        return nil, err
    }
    extraServer, err := extraConfig.complete().New()
    if err != nil {
        return nil, err
    }
    ...
    func (c *ExtraConfig) complete() *completedExtraConfig {
        if c.Addr == "" {
            c.Addr = "127.0.0.1:8081"
        }
        return &completedExtraConfig{c}
    }

In the above code, the configuration is completed by calling the Complete/complete functions, and then an instance of the HTTP/GRPC service is created based on the completed configuration.

Here is a design technique: the complete function returns an instance of type *completedExtraConfig, and when creating the GRPC instance, it calls the New method provided by the completedExtraConfig struct. This design ensures that the GRPC instance we create is based on the completed configuration.

In actual Go project development, we need to provide a mechanism to handle or complete configurations, which is a very useful step in Go project development.

Finally, call the PrepareRun method to prepare for the startup of the HTTP/GRPC server. In the preparation function, we can do various initialization operations, such as initializing the database, installing business-related middleware in Gin, and creating RESTful API routes.

After completing the preparation for the startup of the HTTP/GRPC server, call the Run method to start the HTTP/GRPC service. In the Run method, the GRPC and HTTP services are started separately.

As you can see, the software framework of the whole iam-apiserver is quite clear.

After the service is started, it can handle requests. So next, let’s take a look at the request processing process of iam-apiserver’s REST API.

REST API Request Processing Process of iam-apiserver #

The request processing process of iam-apiserver is also clear and standardized. The specific process is shown in the following diagram:

Combining the above diagram, let’s look at the REST API request processing process of iam-apiserver to help you better understand how iam-apiserver handles HTTP requests.

First, we send a request to the RESTful API interface provided by iam-apiserver through the API call (<HTTP Method> + <HTTP Request Path>).

Next, after receiving the HTTP request, the Gin web framework completes the request authentication through the authentication middleware. iam-apiserver provides two authentication methods: Basic Authentication and Bearer Authentication. After the authentication is successful, the request is processed by a series of middleware that we load, such as Cross-Origin Resource Sharing (CORS), RequestID, and Dump.

Finally, the routing is matched based on <HTTP Method> + <HTTP Request Path>.

For example, let’s say we are requesting the RESTful API POST + /v1/secrets. The Gin Web framework will match the registered controllers based on the HTTP Method and HTTP Request Path. Ultimately, it will match the secretController.Create Controller.

In the Create Controller, we will sequentially perform request parameter parsing, request parameter validation, call the business layer method to create a Secret, handle the return result from the business layer, and finally return the HTTP request result.

iam-apiserver Code Architecture #

The iam-apiserver code design follows a clean architecture, which has the following 5 characteristics:

• Independent of frameworks: This architecture does not rely on powerful software libraries. This allows you to use the framework as a tool rather than being constrained by it.
• Testability: Business rules can be tested without UI, database, web services, or other external elements. In actual development, we decouple these dependencies using mocks.
• Independent of UI: The UI can be easily changed without changing other parts of the system. For example, the web UI can be replaced with a console UI without changing the business rules.
• Independent of the database: You can replace MariaDB with MongoDB, Oracle, Etcd or other databases without binding your business rules to a specific database.
• Independent of external mechanisms: In reality, your business rules can be so simple that they don’t need to know about the external world.

Therefore, based on these constraints, each layer must be independent and testable. The iam-apiserver code architecture is divided into 4 layers: Model Layer (Models), Controller Layer (Controller), Service Layer (Service), and Repository Layer (Repository). From the Controller Layer, to the Service Layer, to the Repository Layer, the level deepens from left to right. The Model Layer is independent of the other layers and can be referenced by other layers. As shown in the figure below:

There are strict import relationships between layers to prevent circular imports. The import relationships are as follows:

• The Model Layer’s package can be imported by the Repository Layer, Service Layer, and Controller Layer.
• The Controller Layer can import packages from the Service Layer and the Repository Layer. It is important to note that unless there is a specific need, the Controller Layer should avoid importing packages from the Repository Layer. The Controller Layer completes its business functions through the Service Layer. This makes the logic of the code clearer and more standardized.
• The Service Layer can import packages from the Repository Layer.

Next, let’s take a detailed look at the functionality of each layer and some important points.

1. Model Layer (Models)

In some software architectures, the Model Layer is also called the Entity Layer. The models are used in each layer and store the structure and methods of objects in this layer. The models in the IAM project are stored in the directory github.com/marmotedu/api/apiserver/v1 and define models such as User, UserList, Secret, SecretList, Policy, PolicyList, and AuthzPolicy, along with their methods. For example:

type Secret struct {
	// May add TypeMeta in the future.
	// metav1.TypeMeta `json:",inline"`

	// Standard object's metadata.
	metav1.ObjectMeta `       json:"metadata,omitempty"`
	Username          string `json:"username"           gorm:"column:username"  validate:"omitempty"`
	SecretID          string `json:"secretID"           gorm:"column:secretID"  validate:"omitempty"`
	SecretKey         string `json:"secretKey"          gorm:"column:secretKey" validate:"omitempty"`

	// Required: true
	Expires     int64  `json:"expires"     gorm:"column:expires"     validate:"omitempty"`
	Description string `json:"description" gorm:"column:description" validate:"description"`
}

The reason the models are stored in the github.com/marmotedu/api project and not in the github.com/marmotedu/iam project is to make these models available to other projects. For example, the IAM models can be imported by the github.com/marmotedu/shippy application. Similarly, the models of the shippy application can also be imported by the IAM project. The import relationship is shown in the following diagram:

The above dependencies are all one-way relationships, with clear dependencies and no circular dependencies.

To add a model definition for “shippy”, you only need to create a new directory in the “api” directory. For example, if there is a “vessel” service in the “shippy” application, the package where the model is located can be github.com/marmotedu/api/vessel.

In addition, the model here can serve as both a database model and a request model for the API (input and output parameters). If we can ensure that the attributes for creating resources, attributes stored in the database for resources, and attributes returned for resources are consistent, we can use the same model. By using the same model, our code can be more concise, maintainable, and developer-efficient. If there are differences in these three attributes, you can create a new model separately to adapt.

1. Repository Layer

The repository layer is used to interact with databases/third-party services and serves as the data engine of the application for inputting and outputting application data. It is important to note that the repository layer only performs CRUD operations on the database/third-party services and does not encapsulate any business logic.

The repository layer is also responsible for selecting the type of database that will be used in the application, such as MySQL, MongoDB, MariaDB, Etcd, etc. Regardless of which database is used, it should be determined at this layer. The repository layer depends on connecting to the database or other third-party services (if any).

This layer also plays a role in data conversion: converting data obtained from the database/microservices into data structures recognizable by the controller and business layers, and converting data formats in the controller and business layers into data formats recognizable by the database or microservices.

The repository layer of iam-apiserver is located in the internal/apiserver/store/mysql directory. The methods inside are used to interact with MariaDB to perform CRUD operations, such as retrieving a secret from the database:

func (s *secrets) Get(ctx context.Context, username, name string, opts metav1.GetOptions) (*v1.Secret, error) {
    secret := &v1.Secret{}
    err := s.db.Where("username = ? and name= ?", username, name).First(&secret).Error
    if err != nil {
        if errors.Is(err, gorm.ErrRecordNotFound) {
            return nil, errors.WithCode(code.ErrSecretNotFound, err.Error())
        }

        return nil, errors.WithCode(code.ErrDatabase, err.Error())
    }

    return secret, nil
}

2. Service Layer

The service layer is mainly used to handle business logic. We can place all business logic processing code in the service layer. The service layer handles requests from the controller, and requests the repository layer to perform CRUD operations as needed. The functions of the service layer are shown in the following diagram:

The service layer of iam-apiserver is located in the internal/apiserver/service directory. The following is a function in the iam-apiserver service layer used to create a secret:

func (s *secretService) Create(ctx context.Context, secret *v1.Secret, opts metav1.CreateOptions) error {
    if err := s.store.Secrets().Create(ctx, secret, opts); err != nil {
        return errors.WithCode(code.ErrDatabase, err.Error())
    }

    return nil
}

As you can see, the service layer ultimately requests the Create method of s.store in the repository layer to save the secret information in the MariaDB database.

3. Controller Layer The controller layer receives HTTP requests and performs operations such as parameter parsing, parameter validation, logic dispatching, and request response. The controller layer will delegate the logic to the business layer, which will handle it and return the result. The returned data will be aggregated and processed in the controller layer before being returned to the requester. The controller layer functions as the implementation of the business routing. The specific process is shown in the following diagram:

Here, I have a suggestion: do not write complex code in the controller layer. If necessary, distribute this code to the business layer or other packages.

The controller layer of iam-apiserver is located in the internal/apiserver/controller directory. Here is the code for creating a secret in the iam-apiserver controller layer:

func (s *SecretHandler) Create(c *gin.Context) {
	log.L(c).Info("create secret function called.")

	var r v1.Secret

	if err := c.ShouldBindJSON(&r); err != nil {
		core.WriteResponse(c, errors.WithCode(code.ErrBind, err.Error()), nil)

		return
	}

	if errs := r.Validate(); len(errs) != 0 {
		core.WriteResponse(c, errors.WithCode(code.ErrValidation, errs.ToAggregate().Error()), nil)

		return
	}

	username := c.GetString(middleware.UsernameKey)

	secrets, err := s.srv.Secrets().List(c, username, metav1.ListOptions{
		Offset: pointer.ToInt64(0),
		Limit:  pointer.ToInt64(-1),
	})
	if err != nil {
		core.WriteResponse(c, errors.WithCode(code.ErrDatabase, err.Error()), nil)

		return
	}

	if secrets.TotalCount >= maxSecretCount {
		core.WriteResponse(c, errors.WithCode(code.ErrReachMaxCount, "secret count: %d", secrets.TotalCount), nil)

		return
	}

	// must reassign username
	r.Username = username

	if err := s.srv.Secrets().Create(c, &r, metav1.CreateOptions{}); err != nil {
		core.WriteResponse(c, err, nil)

		return
	}

	core.WriteResponse(c, nil, r)
}

The above code performs the following operations:

1. Parse HTTP request parameters.
2. Perform parameter validation, and here you can add some business-specific parameter validation, such as secrets.TotalCount >= maxSecretCount.
3. Call the Create method of s.srv in the business layer to create the secret.
4. Return the HTTP request parameters.

Above, we introduced the 4-layer architecture used by iam-apiserver, and next, let’s see how each layer communicates with each other.

Except for the model layer, the controller layer, business layer, and repository layer communicate with each other through interfaces. By communicating through interfaces, on the one hand, the same functionality can support different implementations (i.e., plug-in capabilities), and on the other hand, it makes the code of each layer testable.

Here, let me explain how each layer communicates using the example of a request to create a secret API.

First, let’s see how the controller layer communicates with the business layer.

All secret request processing is handled through methods provided by the SecretController, and creating a secret calls its Create method:

func (s *SecretController) Create(c *gin.Context) {
    ...
	if err := s.srv.Secrets().Create(c, &r, metav1.CreateOptions{}); err != nil {
		core.WriteResponse(c, err, nil)

		return
	}
	...
}

In the Create method, the Create() of s.srv.Secrets() is called to create the secret. s.srv is an interface type defined as follows:

type Service interface {
    Users() UserSrv
    Secrets() SecretSrv
    Policies() PolicySrv
}

type SecretSrv interface {                                                             
    Create(ctx context.Context, secret *v1.Secret, opts metav1.CreateOptions) error    
    Update(ctx context.Context, secret *v1.Secret, opts metav1.UpdateOptions) error            
    Delete(ctx context.Context, username, secretID string, opts metav1.DeleteOptions) error                        
    DeleteCollection(ctx context.Context, username string, secretIDs []string, opts metav1.DeleteOptions) error    
    Get(ctx context.Context, username, secretID string, opts metav1.GetOptions) (*v1.Secret, error)    
    List(ctx context.Context, username string, opts metav1.ListOptions) (*v1.SecretList, error)    
} 

As we can see, the controller layer isolates the specific implementation of the business layer by using the Service interface type provided by the business layer. The Service interface type in the business layer provides the Secrets() method, which returns an instance that implements the SecretSrv interface. In the controller layer, the creation of the secret is completed by calling the Create(ctx context.Context, secret *v1.Secret, opts metav1.CreateOptions) error method of that instance. The controller layer does not need to know how the business layer creates the secret, which means there can be multiple implementations for creating the secret.

This applies the Factory Method Pattern. Service is the factory interface, which includes a series of factory functions for creating concrete business layer objects: Users(), Secrets(), Policies(). With the factory method pattern, it not only hides the details of creating business layer objects but also allows easily adding new business layer objects in the implementation methods of the Service factory interface.

For example, if we want to add a new Template business layer object, which is used to provide some preconfigured policy templates in iam-apiserver, we can add it like this:

type Service interface {
    Users() UserSrv
    Secrets() SecretSrv
    Policies() PolicySrv
    Templates() TemplateSrv
}

func (s *service) Templates() TemplateSrv {
    return newTemplates(s)
}

Next, create a new template.go file:

type TemplateSrv interface {
    Create(ctx context.Context, template *v1.Template, opts metav1.CreateOptions) error
    // Other methods
}

type templateService struct {
    store store.Factory
}

var _ TemplateSrv = (*templateService)(nil)

func newTemplates(srv *service) *TemplateService {
    // more create logic
    return &templateService{store: srv.store}
}

func (u *templateService) Create(ctx context.Context, template *v1.Template, opts metav1.CreateOptions) error {
    // normal code

    return nil
}

As we can see, we have added a new business layer object in the following three steps:

1. In the Service interface definition, we added a new entry: Templates() TemplateSrv.
2. In the service.go file, we added a new function: Templates().
3. Created a new template.go file, which defined the templateService structure and implemented the TemplateSrv interface for it.

As we can see, the code for the newly added Template business object is almost enclosed in the template.go file. Other than adding a factory method Templates() TemplateSrv to the existing Service factory interface, no other invasions were made. This avoids affecting existing businesses.

In actual project development, you may also come up with the following incorrect creation method:

// Incorrect method 1
type Service interface {
    UserSrv
    SecretSrv
    PolicySrv
    TemplateSrv
}

In the above creation method, if we want to create User and Secret, we can only define two different methods: CreateUser and CreateSecret, which is far less elegant than providing the same name Create method in the respective domains of User and Secret.

The factory interface in the IAM project also uses the factory method pattern in other places, such as the Factory factory interface.

Now let’s see how the business layer and repository layer communicate.

The business layer and repository layer also communicate through interfaces. For example, the code for creating a secret in the business layer is as follows:

func (s *secretService) Create(ctx context.Context, secret *v1.Secret, opts metav1.CreateOptions) error {
    if err := s.store.Secrets().Create(ctx, secret, opts); err != nil {
        return errors.WithCode(code.ErrDatabase, err.Error())
    }

    return nil
}

The Create method calls the s.store.Secrets().Create() method to save the secret to the database. s.store is an interface type defined as follows:

type Factory interface {
    Users() UserStore
    Secrets() SecretStore
    Policies() PolicyStore
    Close() error
}

The communication between the business layer and the repository layer is similar to the implementation of the communication between the control layer and the business layer, so we won’t go into detail here.

So far, we have learned that the control layer, business layer, and repository layer communicate through interfaces. Communicating through interfaces has a benefit, which is making each layer testable. Next, let’s see how to test the code in each layer. Since the testing of Go code will be covered in detail in Lesson 38 and Lesson 39, I will only briefly introduce the test ideas here.

1. Model Layer

Since the model layer doesn’t depend on any other layer, we only need to test the defined structures and their functions and methods.

2. Control Layer

The control layer depends on the business layer, which means that this layer requires the support of the business layer for testing. You can use golang/mock to mock the business layer, and test cases can refer to TestUserController_Create.

3. Business Layer

Since this layer depends on the repository layer, it means that this layer requires the support of the repository layer for testing. We have two ways to mock the repository layer:

• Mock the repository layer using golang/mock.
• Develop a fake repository layer ourselves.

For test cases that use golang/mock, you can refer to Test_secretService_Create.

You can refer to the fake repository layer for test cases that use the fake repository layer. The test case that uses this fake repository layer for testing is Test_userService_List.

4. Repository Layer

The repository layer depends on the database and may depend on third-party services if other microservices are called. We can use sqlmock to simulate the database connection and httpmock to simulate HTTP requests.

Summary #

In this lecture, I mainly introduced the functions and usage methods of iam-apiserver, as well as its code implementation. iam-apiserver is a web service that provides REST APIs to perform CRUD operations on three types of REST resources: users, keys, and policies. We can use tools like cURL and Insomnia to make REST API requests.

iam-apiserver includes three types of configurations: options configuration, application configuration, and HTTP/GRPC service configuration. These three configurations are used to build command-line parameters, applications, and HTTP/GRPC services, respectively.

When iam-apiserver starts, it first constructs the application framework, then sets the application options, initializes the application, creates the configuration and instance of the HTTP/GRPC service, and finally starts the HTTP/GRPC service.

After the service is started, it can receive HTTP requests. An HTTP request is first authenticated, then processed by registered middleware. Next, it is matched to the handling function based on (HTTP Method, HTTP Request Path). In the handling function, the request parameters are parsed, validated, and processed by the business logic function, and finally the request result is returned.

iam-apiserver adopts a concise architecture, where the entire application is divided into four layers: the model layer, the controller layer, the service layer, and the repository layer. The model layer stores the structure and methods of stored objects, the repository layer interacts with databases/third-party services for CRUD operations, the service layer is mainly used to complete business logic processing, and the controller layer receives HTTP requests, performs parameter parsing, validation, logic distribution, and request response operations. The controller layer, service layer, and repository layer communicate through interfaces. This allows the same functionality to be supported by different implementations and makes the code of each layer testable.

Post-class Exercises #

1. Both iam-apiserver and iam-authz-server provide REST API services. Read their source code and find out how iam-apiserver and iam-authz-server share the related REST API code.

2. Think about whether the service construction method of iam-apiserver can be abstracted into a template (Go package)? If so, how to abstract it?

Feel free to leave me a message in the comments section to discuss and exchange ideas. See you in the next lesson.