01 Iam System Overview What We Need to Implement in Go Project

01 IAM System Overview What We Need to Implement in Go Project #

Hello, I am Kong Lingfei. Starting today, we will enter the pre-course preparation stage. I will take three lessons to explain clearly what our practical project IAM application looks like, what it can do, and how to deploy it on a Linux server. Let’s start by removing the basic obstacles together so that you can learn the subsequent courses more easily.

In today’s lesson, I will first talk about why we choose IAM application, what functions it can achieve, and its architecture and usage process.

Project Background: Why choose IAM system as a practical project? #

When developing Go projects, security is an unavoidable topic. How to ensure the security of Go applications is a problem that every developer needs to solve. Although the security of Go applications involves many aspects, it can be broadly divided into the following two categories:

  • Security of the service itself: To ensure the security of the service, it is necessary to prevent unauthorized users from accessing the service. This can be achieved at the server level and the software level. Server-level security can be ensured from the bottom layer through physical isolation, network isolation, firewall, and other technologies. Software-level security can be enhanced through HTTPS, user authentication, and other means. Server-level security is generally taken care of by the operations team, while software-level security needs to be ensured by developers.
  • Security of service resources: There are many resources within the service, and in order to prevent unauthorized access, developers need to prevent User A from accessing User B’s resources, which requires authorized access to resources. Usually, we can authorize resources through a resource authorization system.

In summary, to ensure the security of Go applications, we need to authenticate access and authorize resources. So how do we implement access authentication and resource authorization?

Authentication is not complicated, and we can implement it through JSON Web Token (JWT) authentication. Authorization, on the other hand, is more complex, and its complexity encompasses many Go development skills. Therefore, in this column, I will upgrade the implementation of authentication and authorization to an IAM system. By explaining its construction process, I will clarify the entire process of Go project development.

What is an IAM system? #

IAM (Identity and Access Management) system is a web service written in Go language that provides access control services to third-party users.

The problem that IAM system can help users solve is: Under specific conditions, who can/cannot perform which actions on which resources (Who is able to do what on something given some context), which refers to the function of resource authorization.

Note: In the future, when referring to IAM system or IAM, we are referring to the IAM application.

So, how does the IAM system perform resource authorization? Below, let’s take a look at how it works through the process of resource authorization in the IAM system. The entire process can be divided into 4 steps.

  1. Users need to provide information such as nickname, password, email, and phone number to register and log in to the IAM system. Here, the username and password are used as the unique identity to access the IAM system and complete authentication.
  2. Since accessing the IAM’s resource authorization interface is authenticated through keys (secretID/secretKey), users need to create their own key resources in IAM.
  3. Because IAM completes authorization through authorization policies, users need to create authorization policies in IAM.
  4. Request the authorization interface provided by IAM. IAM will determine whether an authorization request is allowed based on the user’s request content and authorization policy.

We can see that in the above process, IAM uses 3 types of system resources: users, keys, and policies, which are mapped to 3 RESTful resources in program design:

  • Users: Implement operations such as creating, deleting, modifying, querying, changing passwords, and batch modifications on users.
  • Keys: Implement operations such as creating, deleting, modifying, and querying keys.
  • Policies: Implement operations such as creating, deleting, modifying, querying, and batch deleting policies.

What does the architecture of IAM system look like? #

After understanding the functionality of IAM, let’s talk about the architecture of the IAM system in detail. The architecture diagram is as follows:

In general, the IAM architecture includes 9 major components and 3 major databases. I have summarized these components and their functions in the table below. Among them, we mainly remember the functions of 5 core components, including iam-apiserver, iam-authz-server, iam-pump, marmotedu-sdk-go, and iamctl. There are also 3 databases - Redis, MySQL, and MongoDB.

In addition, the three databases used for storing data in the IAM system are described as follows:

Understanding the Architecture through Workflow #

Just seeing the system architecture diagram and the explanation of core functions, you may not yet have a clear understanding of how the entire system collaborates to ultimately complete resource authorization. Therefore, next, we will further deepen your understanding of the IAM architecture by explaining in detail the usage workflow and implementation details of the IAM system. In general, we can use 4 steps to use the core functions of the IAM system.

Step 1: Creating platform resources.

Users use the iam-webconsole (RESTful API) or iamctl (sdk marmotedu-sdk-go) client to request the RESTful API interface provided by iam-apiserver to complete the addition, deletion, modification, and query of users, keys, and authorization policies. iam-apiserver will persistently store these resource data in the MySQL database. Also, to ensure communication security, the communication between the client and the server is done through HTTPS.

Step 2: Request API for resource authorization.

Users can request the /v1/authz interface provided by iam-authz-server for resource authorization. Access to the /v1/authz interface requires key authentication. After authentication, the /v1/authz interface will query the authorization policy to determine whether the resource request is allowed.

In order to improve the performance of the /v1/authz interface, iam-authz-server caches the key and policy information in memory for fast query. How are the key and policy information cached?

First, iam-authz-server caches the key and authorization policy information in memory by calling the gRPC interface provided by iam-apiserver. At the same time, in order to keep the cache information in memory consistent with the information in iam-apiserver, when a key or policy is updated in iam-apiserver, iam-apiserver will send a PolicyChanged or SecretChanged message to a specific Redis Channel (which iam-authz-server also subscribes to). In this way, when iam-authz-server detects a new message, it will retrieve and parse the message, determine whether to re-invoke the gRPC interface to obtain the key and authorization policy information, and update it to memory.

Step 3: Authorization log data analysis.

iam-authz-server will report the authorization logs to the Redis cache, and then the iam-pump component will consume these authorization logs asynchronously and save the processed data to MongoDB for querying by the operating system iam-operating-system.

One thing to note here: iam-authz-server stores the authorization logs in the Redis high-performance key-value database to minimize write latency. Not storing them in memory is because we cannot predict the volume of authorization logs. When the volume is large, it is likely to exhaust the memory and cause service interruption.

Step 4: Displaying authorization data on the operating platform.

iam-operating-system is the operating system of IAM. It can query and display operational data by querying MongoDB, such as the number of authorizations/failures for a user, authorization information when authorization fails, etc. In addition, we can also use iam-operating-system to call iam-apiserver services for operational and management purposes. For example, view the authorization policy of a user from the perspective of a super administrator for troubleshooting purposes, adjust the maximum number of keys a user can create, or use the whitelist to exempt a user from the limit on the number of keys, and so on.

IAM Software Architecture Patterns #

When designing software, the first thing to do is to choose a software architecture pattern, which has a significant impact on the subsequent development and maintenance costs of the software. Therefore, here I will briefly discuss two of the most commonly used software architecture patterns: the frontend-backend separation architecture and the MVC architecture.

Front-end and Back-end Separation Architecture #

Since the IAM system adopts a front-end and back-end separation architecture, let’s take the IAM operating system (iam-operating-system) as an example to explain this architecture in detail. Generally speaking, the operational system can have more or less functions. For operational systems with complex functions, we can adopt a front-end and back-end separation architecture. Wherein, the front-end is responsible for displaying pages, loading and rendering data, and the back-end only returns the data needed by the front-end.

The front-end and back-end separation architecture of iam-operating-system is shown in the following figure.

After adopting the front-end and back-end separation architecture, when you request the front-end ‘ops-webconsole’ through the browser, ‘ops-webconsole’ will first request the static file server to load static files, such as HTML, CSS, and JavaScript. Then it will execute JavaScript, request back-end data through load balancing, and finally render the data returned by the back-end to the front-end page.

Using the front-end and back-end separation architecture, allowing communication between the front-end and back-end through RESTful APIs, has the following 5 benefits:

  • It allows front-end and back-end developers to focus on their own functional development, so that professionals can do professional work to improve code quality and development efficiency.
  • Front-end and back-end development and deployment can be done in parallel, which improves development and release efficiency and speeds up product iteration.
  • The separation of front-end and back-end components and code clarifies responsibilities, enhances code maintainability and readability, reduces the probability of bugs caused by code changes, and also enables quick bug localization.
  • Front-end JavaScript can handle data from the back-end, reducing the load on the back-end server.
  • Front-end or back-end can be horizontally scaled based on actual needs to save costs.

MVC Architecture #

However, if the operational system has fewer functions, adopting a front-end and back-end separation framework may bring more disadvantages than advantages. For example, maintaining 2 components simultaneously in front-end and back-end separation leads to more complex deployment. Additionally, the separation of front-end and back-end personnel increases the communication cost to a certain extent. Moreover, because the code also needs to implement the logic of front-end and back-end interaction, it introduces certain development efforts.

At this time, we can try using the MVC software architecture directly. The MVC architecture is shown in the following figure.

The complete name of MVC is Model View Controller. It is an architectural pattern, divided into three layers: Model, View, and Controller. The functions of each layer are as follows:

  • View: Provides the user interface for handling data display.
  • Controller: Based on the instructions input from the View layer by the user, selects data from the Model layer, and then performs corresponding operations to produce the final result.
  • Model: The part of the application responsible for handling data logic.

The advantage of the MVC architecture is that after separating the View layer and Model layer through the Controller layer, when we modify the View layer code, we do not need to recompile the code of the Controller layer and Model layer. Similarly, if there are changes in business processes, we only need to modify the code of the Model layer. In actual development, in order to achieve better UI effects, the View layer often needs to be changed. However, through the MVC architecture, when changing the View layer, we don’t need to make any changes to the code of the business logic layer. This not only reduces risk but also improves code modification and release efficiency.

In addition, there is another software development architecture called the three-tier architecture, which includes the UI layer, BLL layer, and DAL layer. Among them, the UI layer represents the user interface, the BLL layer represents the business logic, and the DAL layer represents the data access. In actual development, many people treat MVC as a three-tier architecture. For example, many people like to put the software’s business logic in the Controller layer and put the code for database access operations in the Model layer, and finally put the software code in the View layer. In this way, MVC architecture is forcibly transformed into a pseudo three-tier architecture.

Such code is neither fish nor fowl, and it also loses the core advantages of the three-tier architecture and MVC architecture, which is to decouple the Model layer and View layer through the Controller layer, making the code easier to maintain and scale. Therefore, in actual development, we should also pay attention to following the development specifications of the MVC architecture and exert the core value of MVC.

Summary #

A good Go application must ensure the security of the application, which can be achieved through authentication and authorization. Therefore, authentication and authorization are essential functionalities that must be implemented when developing a Go project. To help you implement these two functionalities and learn Go project development at the same time, I have upgraded them to an IAM system. By explaining how to develop an IAM system, I will teach you how to develop Go projects.

Since the subsequent learning revolves around the IAM system, in this lesson, we need to focus on understanding the functions, architecture, and workflow of IAM. We can understand this through a four-step process.

First, users can register and log in to the system by calling the RESTful API interface provided by iam-apiserver. They can also call the interface to create keys and authorization policies.

After creating the key pair and authorization policies, IAM can authorize resources by calling the authorization interface of iam-authz-server. Specifically, iam-authz-server obtains the key and authorization policy information stored in iam-apiserver through the gRPC interface. After completing authentication through JWT, it uses the ory/ladon package to authorize resources.

Next, the iam-pump component asynchronously consumes data from Redis and persistently stores it in MongoDB for display in the iam-operating-system operation platform.

Finally, IAM-related products and developers can use the iam-operating-system to view the usage of the IAM system and perform operational analysis. For example, they can view the authorization/failure times of a user and the authorization information when authorization fails.

In addition, to improve development and access efficiency, IAM provides the marmotedu-sdk-go SDK and the iamctl command-line tool. Both tools access the RESTful interface provided by IAM through the HTTPS protocol.

Exercises #

  1. What skills do you often use when developing Go projects? Are there any skills that IAM does not cover?
  2. In the projects you have worked on, which ones follow a frontend-backend separation architecture and which ones follow MVC architecture? Do you think the chosen architecture for the projects is reasonable?

Looking forward to hearing your thoughts and sharing in the comments section. See you in the next lesson!