34 Sdk Design Iam Project Go Sdk Design and Implementation

34 SDK Design IAM Project Go SDK Design and Implementation #

Hello, I’m Kong Lingfei.

In the previous lesson, I introduced the SDK design approach commonly adopted by public cloud providers. However, there are also some excellent SDK design approaches, such as the design approach used by Kubernetes’ client-go SDK. The IAM project references client-go and has also implemented an SDK in the client-go style: marmotedu-sdk-go.

Compared to the SDK design approach introduced in Lesson 33, the client-go style SDK has the following advantages:

  • It extensively uses the Go interface feature, decoupling the interface definition from the implementation and supporting multiple implementation methods.
  • The interface call hierarchy matches the resource hierarchy, making the calling process more user-friendly.
  • It supports multiple versions coexisting.

Therefore, I highly recommend using marmotedu-sdk-go. Next, let’s take a look at how marmotedu-sdk-go is designed and implemented.

Design of marmotedu-sdk-go #

Compared to medu-sdk-go, the design and implementation of marmotedu-sdk-go are more complex, but it is more powerful and provides a better user experience.

Here, let’s start with an example of using the SDK to call the iam-authz-server /v1/authz interface. The code is saved in the marmotedu-sdk-go/examples/authz_clientset/main.go file:

package main

import (
	"context"
	"flag"
	"fmt"
	"path/filepath"

	"github.com/ory/ladon"

	metav1 "github.com/marmotedu/component-base/pkg/meta/v1"
	"github.com/marmotedu/component-base/pkg/util/homedir"

	"github.com/marmotedu/marmotedu-sdk-go/marmotedu"
	"github.com/marmotedu/marmotedu-sdk-go/tools/clientcmd"
)

func main() {
	var iamconfig *string
	if home := homedir.HomeDir(); home != "" {
		iamconfig = flag.String(
			"iamconfig",
			filepath.Join(home, ".iam", "config"),
			"(optional) absolute path to the iamconfig file",
		)
	} else {
		iamconfig = flag.String("iamconfig", "", "absolute path to the iamconfig file")
	}
	flag.Parse()

	// use the current context in iamconfig
	config, err := clientcmd.BuildConfigFromFlags("", *iamconfig)
	if err != nil {
		panic(err.Error())
	}

	// create the clientset
	clientset, err := marmotedu.NewForConfig(config)
	if err != nil {
		panic(err.Error())
	}

	request := &ladon.Request{
		Resource: "resources:articles:ladon-introduction",
		Action:   "delete",
		Subject:  "users:peter",
		Context: ladon.Context{
			"remoteIP": "192.168.0.5",
		},
	}

	// Authorize the request
	fmt.Println("Authorize request...")
	ret, err := clientset.Iam().AuthzV1().Authz().Authorize(context.TODO(), request, metav1.AuthorizeOptions{})
	if err != nil {
		panic(err.Error())
	}

	fmt.Printf("Authorize response: %s.\n", ret.ToString())
}

In the above code example, the following operations are included:

  • First, call the BuildConfigFromFlags function to create the SDK’s configuration instance config.
  • Then, call marmotedu.NewForConfig(config) to create the client clientset for the IAM project.
  • Finally, call the following code to request the /v1/authz interface and execute the resource authorization request:
ret, err := clientset.Iam().AuthzV1().Authz().Authorize(context.TODO(), request, metav1.AuthorizeOptions{})
if err != nil {
	panic(err.Error())
}

fmt.Printf("Authorize response: %s.\n", ret.ToString())

The format of the call is project-client.application-client.service-client.resource-name.interface.

Therefore, the above code calls the API interface of the resource by creating project-level client, application-level client, and service-level client. Next, let’s see how to create these clients.

Design of marmotedu-sdk-go clients #

Before discussing client creation, let’s first look at the design approach of the clients.

Go projects are organized hierarchically: Project -> Application -> Service. marmotedu-sdk-go well reflects this hierarchical relationship, making SDK calls easier to understand and use. The hierarchical relationship of marmotedu-sdk-go is shown in the following diagram:

marmotedu-sdk-go defines three types of interfaces, representing project, application, and service-level API interfaces respectively:

// Project-level interface
type Interface interface {
    Iam() iam.IamInterface
    Tms() tms.TmsInterface
}

// Application-level interface
type IamInterface interface {
    APIV1() apiv1.APIV1Interface
    AuthzV1() authzv1.AuthzV1Interface
}

// Service-level interface
type APIV1Interface interface {
    RESTClient() rest.Interface
    SecretsGetter
    UsersGetter
}

    PoliciesGetter
}

// Client for resource level
type SecretsGetter interface {
    Secrets() SecretInterface
}

// Interface definition for the resource
type SecretInterface interface {
    Create(ctx context.Context, secret *v1.Secret, opts metav1.CreateOptions) (*v1.Secret, error)
    Update(ctx context.Context, secret *v1.Secret, opts metav1.UpdateOptions) (*v1.Secret, error)
    Delete(ctx context.Context, name string, opts metav1.DeleteOptions) error
    DeleteCollection(ctx context.Context, opts metav1.DeleteOptions, listOpts metav1.ListOptions) error
    Get(ctx context.Context, name string, opts metav1.GetOptions) (*v1.Secret, error)
    List(ctx context.Context, opts metav1.ListOptions) (*v1.SecretList, error)
    SecretExpansion
}

Interface represents the project-level interface, which includes two applications: Iam and Tms. IamInterface represents the application-level interface, which includes the API (iam-apiserver) and authz (iam-authz-server) service-level interfaces. In the API and authz services, each service includes the CURD interface of the REST resource.

The marmotedu-sdk-go supports different versions of API interfaces using the naming convention XxxV1. The benefit of this approach is that it allows calling different versions of the same API interface in the program. For example:

clientset.Iam().AuthzV1().Authz().Authorize() and clientset.Iam().AuthzV2().Authz().Authorize() invoke the /v1/authz and /v2/authz versions of the API interface, respectively.

This relationship can also be reflected in the directory structure. The marmotedu-sdk-go directory is designed as follows (only listing some important files):

├── examples                        # Contains examples of SDK usage
├── Makefile                        # Manages the SDK source code, static code checking, code formatting, testing, adding copyright information, etc.
├── marmotedu
│   ├── clientset.go                # Implementation of clientset, which contains multiple applications and multiple service-level API interfaces
│   ├── fake                        # Fake implementation of clientset, mainly used for unit testing
│   └── service                     # Classified by application, contains the specific implementation of API interfaces in each service of the application
│       ├── iam                     # Implementation of IAM application API interfaces, including multiple services
│       │   ├── apiserver           # API interfaces of the apiserver service in the IAM application, including multiple versions
│       │   │   └── v1              # apiserver v1 version API interface
│       │   ├── authz               # API interfaces of the authz service in the IAM application
│       │   │   └── v1              # authz service v1 version interface
│       │   └── iam_client.go       # Client of the IAM application, containing clients for the apiserver and authz services
│       └── tms                     # Implementation of TMS application API interfaces
├── pkg                             # Contains some shared packages that can be exposed externally
├── rest                            # Underlying implementation of HTTP requests
├── third_party                     # Contains modified third-party packages, such as gorequest
└── tools
    └── clientcmd                   # Contains some functions to help create rest.Config configurations

Each type of client can be created in a similar way:

config, err := clientcmd.BuildConfigFromFlags("", "/root/.iam/config")
clientset, err := xxxx.NewForConfig(config)

/root/.iam/config is the configuration file, which includes the addresses and authentication information of the service. The BuildConfigFromFlags function loads the configuration file, creates and returns a configuration variable of type rest.Config, and creates the necessary client through the xxxx.NewForConfig function. xxxx represents the client package at that level, such as iam, tms.

The marmotedu-sdk-go client defines three types of interfaces, which brings two benefits.

First, it standardizes the format of API interface calls and makes the hierarchy clear, making API interface calls more clear and memorable.

Second, it allows clients to be flexibly selected according to needs. For example, if both the iam-apiserver and iam-authz-server interfaces need to be used in service A, an application-level client IamClient can be created, and then different services can be called by using iamclient.APIV1() and iamclient.AuthzV1().

Next, let’s see how to create clients at the three different levels.

Project-level client creation #

The client implementation corresponding to Interface is Clientset, which is located in the package marmotedu-sdk-go/marmotedu. The clientset can be created as follows:

config, err := clientcmd.BuildConfigFromFlags("", "/root/.iam/config")
clientset, err := marmotedu.NewForConfig(config)

The calling format is clientset.application.service.resourceName.interface, for example:

rsp, err := clientset.Iam().AuthzV1().Authz().Authorize()

Refer to the example at marmotedu-sdk-go/examples/authz_clientset/main.go.

Application-level client creation #

The client implementation corresponding to IamInterface is IamClient, which is located in the package marmotedu-sdk-go/marmotedu/service/iam. The IamClient can be created as follows:

config, err := clientcmd.BuildConfigFromFlags("", "/root/.iam/config")
iamclient, err := iam.NewForConfig(config)

The calling format is iamclient.service.resourceName.interface, for example:

rsp, err := iamclient.AuthzV1().Authz().Authorize()

Refer to the example at marmotedu-sdk-go/examples/authz_iam/main.go.

Service-level client creation #

The client implementation corresponding to AuthzV1Interface is AuthzV1Client, which is located in the package marmotedu-sdk-go/marmotedu/service/iam/authz/v1. The AuthzV1Client can be created as follows:

config, err := clientcmd.BuildConfigFromFlags("", "/root/.iam/config")
client, err := v1.NewForConfig(config)

The calling format is client.resourceName.interface, for example:

rsp, err := client.Authz().Authorize()

Refer to the example at marmotedu-sdk-go/examples/authz/main.go.

Above, I introduced how to create clients in marmotedu-sdk-go. Now, let’s take a look at how these clients actually execute REST API requests.

Implementation of marmotedu-sdk-go #

The implementation of marmotedu-sdk-go is similar to medu-sdk-go, and it adopts a layered structure divided into API layer and base layer. The diagram below shows the structure:

RESTClient is the core of the entire SDK. RESTClient utilizes the Request module to build HTTP request methods, paths, body, and authentication information. The Request module ultimately utilizes methods provided by the gorequest package to perform HTTP POST, PUT, GET, DELETE requests, retrieve the HTTP response, and parse it into the specified structure. RESTClient provides methods such as Post(), Put(), Get(), Delete() for clients to make HTTP requests.

marmotedu-sdk-go provides two types of clients: RESTClient client and clients based on RESTClient.

  • RESTClient: A raw type of client that can send direct HTTP requests by specifying the HTTP method, path, and parameters, as shown in the example client.Get().AbsPath("/version").Do().Into().
  • Clients wrapped based on RESTClient: Examples include AuthzV1Client, APIV1Client, which execute requests for specific REST resources or API interfaces, making it convenient for developers to use.

Next, let’s take a closer look at how to create a RESTClient client and the implementation of the Request module.

RESTClient Client Implementation #

I have implemented the RESTClient client in the following two steps.

Step 1: Create a variable of type rest.Config.

The BuildConfigFromFlags function creates a variable of type rest.Config by loading a YAML format configuration file. The content of the loaded YAML format configuration file is as follows:

apiVersion: v1
user:
  #token: # JWT Token
  username: admin # iam username
  password: Admin@2020 # iam password
  #secret-id: # Secret ID
  #secret-key: # Secret Key
  client-certificate: /home/colin/.iam/cert/admin.pem # Path to the client certificate file for TLS
  client-key: /home/colin/.iam/cert/admin-key.pem # Path to the client key file for TLS
  #client-certificate-data:
  #client-key-data:

server:
  address: https://127.0.0.1:8443 # iam api-server address
  timeout: 10s # Timeout for requesting api-server
  #max-retries: # Maximum number of retries, default is 0
  #retry-interval: # Retry interval, default is 1s
  #tls-server-name: # TLS server name
  #insecure-skip-tls-verify: # If set true, it means skipping TLS security verification and makes the HTTPS connection insecure
  certificate-authority: /home/colin/.iam/cert/ca.pem # Path to the CA cert file for authorization
  #certificate-authority-data:

In the configuration file, we can specify the service address, username/password, secret, TLS certificate, timeout, retry count, and other information.

The creation method is as follows:

config, err := clientcmd.BuildConfigFromFlags("", *iamconfig)
if err != nil {
    panic(err.Error())
}

In the above code, *iamconfig is the path to the YAML format configuration file. In the BuildConfigFromFlags function, the LoadFromFile function is called to parse the YAML configuration file. LoadFromFile ultimately parses the YAML format configuration file using yaml.Unmarshal.

Step 2: Create a RESTClient client based on the rest.Config type variable.

Create the RESTClient client using the RESTClientFor function:

func RESTClientFor(config *Config) (*RESTClient, error) {
    ...
    baseURL, versionedAPIPath, err := defaultServerURLFor(config)
    if err != nil {
        return nil, err
    }

    // Get the TLS options for this client config
    tlsConfig, err := TLSConfigFor(config)
    if err != nil {
        return nil, err
    }

    // Only retry when get a server side error.
    client := gorequest.New().TLSClientConfig(tlsConfig).Timeout(config.Timeout).
        Retry(config.MaxRetries, config.RetryInterval, http.StatusInternalServerError)
    // NOTICE: must set DoNotClearSuperAgent to true, or the client will clean header befor http.Do
    client.DoNotClearSuperAgent = true

    ...

    clientContent := ClientContentConfig{
        Username:           config.Username,
        Password:           config.Password,
        SecretID:           config.SecretID,
        SecretKey:          config.SecretKey,
        ...
    }

    return NewRESTClient(baseURL, versionedAPIPath, clientContent, client)
}

}

The RESTClientFor function calls defaultServerURLFor(config) to generate the basic HTTP request path: baseURL=<http://127.0.0.1:8080, versionedAPIPath=/v1. Then, the TLS configuration is generated by the TLSConfigFor function and a gorequest client is created by calling gorequest.New(), with the client configuration information saved in a variable. Finally, the NewRESTClient function is called to create a RESTClient client.

The RESTClient client provides the following methods for the caller to complete the HTTP request:

func (c *RESTClient) APIVersion() scheme.GroupVersion
func (c *RESTClient) Delete() *Request
func (c *RESTClient) Get() *Request
func (c *RESTClient) Post() *Request
func (c *RESTClient) Put() *Request
func (c *RESTClient) Verb(verb string) *Request

It can be seen that RESTClient provides the Delete, Get, Post, and Put methods, which are used to execute the HTTP DELETE, GET, POST, and PUT methods, respectively. The Verb method allows the HTTP method to be specified flexibly. These methods all return variables of type Request. Variables of type Request provide methods to complete specific HTTP requests, for example:

type Response struct {
    Allowed bool   `json:"allowed"`
    Denied  bool   `json:"denied,omitempty"`
    Reason  string `json:"reason,omitempty"`
    Error   string `json:"error,omitempty"`
}

func (c *authz) Authorize(ctx context.Context, request *ladon.Request, opts metav1.AuthorizeOptions) (result *Response, err error) {
    result = &Response{}                              
    err = c.client.Post().
        Resource("authz").
        VersionedParams(opts).
        Body(request).
        Do(ctx).
        Into(result)

    return
}

In the above code, c.client is the RESTClient client, and by calling the Post method of the RESTClient client, a variable of type *Request is returned.

The variable of type *Request provides the Resource and VersionedParams methods to construct the path “/v1/authz” in the HTTP URL; the Body method specifies the HTTP request body.

At this point, we have constructed the parameters required for the HTTP request: the HTTP method, the request URL, and the request body. Therefore, the Do method can be called to execute the HTTP request and save the return result in the passed-in result variable through the Into method.

Request Module Implementation #

The methods of the RESTClient client return variables of type Request, which provide a series of methods to build HTTP request parameters and execute HTTP requests.

Therefore, the Request module can be understood as the lowest-level communication layer. Let’s take a look at how the Request module specifically completes the HTTP request.

First, let’s look at the definition of the Request structure:

type RESTClient struct {            
    // base is the root URL for all invocations of the client    
    base *url.URL    
    // group stand for the client group, eg: iam.api, iam.authz                                        
    group string                                                                                               
    // versionedAPIPath is a path segment connecting the base URL to the resource root    
    versionedAPIPath string                                                                    
    // content describes how a RESTClient encodes and decodes responses.    
    content ClientContentConfig    
    Client  *gorequest.SuperAgent    
}

type Request struct {
	c *RESTClient

	timeout time.Duration

	// generic components accessible via method setters
	verb       string
	pathPrefix string
	subpath    string
	params     url.Values
	headers    http.Header

	// structural elements of the request that are part of the IAM API conventions
	// namespace    string
	// namespaceSet bool
	resource      string
	resourceName  string
	subresource   string

	// output
	err  error
	body interface{}
}

Now let’s look at the methods provided by the Request structure:

func (r *Request) AbsPath(segments ...string) *Request
func (r *Request) Body(obj interface{}) *Request
func (r *Request) Do(ctx context.Context) Result

func (r *Request) Do() (*Response, error) {
    // 构建HTTP请求
    httpRequest, err := r.buildHTTPClientRequest()
    if err != nil {
        return nil, err
    }

    // 执行HTTP请求
    httpResponse, err := r.c.client.Do(httpRequest)
    if err != nil {
        return nil, err
    }

    // 构建Response对象
    response := &Response{
        StatusCode: httpResponse.StatusCode,
    }
    response.rawResponseData, err = ioutil.ReadAll(httpResponse.Body)
    if err != nil {
        return nil, err
    }

    return response, nil
}

func (r *Request) Do(ctx context.Context) Result {
    client := r.c.Client
    client.Header = r.headers

    if r.timeout > 0 {
        var cancel context.CancelFunc
        ctx, cancel = context.WithTimeout(ctx, r.timeout)

        defer cancel()
    }

    client.WithContext(ctx)

    resp, body, errs := client.CustomMethod(r.verb, r.URL().String()).Send(r.body).EndBytes()
    if err := combineErr(resp, body, errs); err != nil {
        return Result{
            response: &resp,
            err:      err,
            body:     body,
        }
    }

    decoder, err := r.c.content.Negotiator.Decoder()
    if err != nil {
        return Result{
            response: &resp,
            err:      err,
            body:     body,
            decoder:  decoder,
        }
    }

    return Result{
        response: &resp,
        body:     body,
        decoder:  decoder,
    }
}

In the Do method, the values of various fields in the Request structure variable are used to execute an HTTP request using client.CustomMethod. client is a client of type *gorequest.SuperAgent.

Step 5: Saving the HTTP response result

The HTTP response result is saved by using the Into method in the Request structure:

func (r Result) Into(v interface{}) error {
    if r.err != nil {
        return r.Error()
    }

    if r.decoder == nil {
        return fmt.Errorf("serializer doesn't exist")
    }

    if err := r.decoder.Decode(r.body, &v); err != nil {
        return err
    }

    return nil
}

r.body is set in the Do method after executing the HTTP request, and its value is the body of the HTTP response.

Request Authentication #

Next, let’s introduce another core feature of the marmotedu-sdk-go: request authentication.

marmotedu-sdk-go supports two authentication methods:

  • Basic authentication: This involves adding Authorization: Basic xxxx to the request.
  • Bearer authentication: This involves adding Authorization: Bearer xxxx to the request. This method supports specifying the JWT token directly or generating a JWT token automatically by specifying a secret key pair.

I have explained Basic authentication and Bearer authentication in Lesson 25, which you can refer to.

The authentication header is specified when the RESTClient client sends an HTTP request, and the specific implementation is located in the NewRequest function:

switch {
    case c.content.HasTokenAuth():
        r.SetHeader("Authorization", fmt.Sprintf("Bearer %s", c.content.BearerToken))
    case c.content.HasKeyAuth():
        tokenString := auth.Sign(c.content.SecretID, c.content.SecretKey, "marmotedu-sdk-go", c.group+".marmotedu.com")
        r.SetHeader("Authorization", fmt.Sprintf("Bearer %s", tokenString))
    case c.content.HasBasicAuth():
        // TODO: get token and set header
        r.SetHeader("Authorization", "Basic "+basicAuth(c.content.Username, c.content.Password))
}

The above code automatically determines which authentication method to use based on the configuration information.

Summary #

In this lecture, I introduced the approach of implementing a Kubernetes client-go style SDK. Compared to the SDK designs of public cloud vendors, the client-go style SDK design has many advantages.

When designing the marmotedu-sdk-go, I implemented three types of clients through interfaces: project-level clients, application-level clients, and service-level clients. Developers can create client types according to their needs.

In marmotedu-sdk-go, a RESTClient type client is created through RESTClientFor. RESTClient, in turn, uses the modules such as Request to build the HTTP request method, request path, request body, and authentication information. The Request module ultimately uses the methods provided by the gorequest package to complete HTTP POST, PUT, GET, DELETE requests, retrieve HTTP response results, and parse them into specified structures. RESTClient provides methods such as Post(), Put(), Get(), Delete() to the client for completing the HTTP requests.

Exercises #

  1. Read the source code of defaultServerURLFor and think about how it constructs the request URL http://iam.api.marmotedu.com:8080 and the API version /v1.

  2. Use the gorequest package to write an example that can execute the following HTTP request:

curl -XPOST http://example.com/v1/user -d '{"username":"colin","address":"shenzhen"}'

Feel free to leave a comment and discuss with me. See you in the next lesson.