Special Delivery IAM Troubleshooting Guide #

Hello, I’m Kong Lingfei.

Today we have an additional episode for a special broadcasting. During the deployment and usage of IAM, it is inevitable to encounter some exceptions (also known as faults or issues). At this time, it is necessary for us to be able to locate and fix these faults. Here, I have summarized some troubleshooting methods for IAM, as well as solutions to common faults, for your reference.

How to Troubleshoot? #

Firstly, we need to identify the problem and then locate it. We may need to go through multiple rounds of analysis and debugging to identify the root cause of the problem and finally resolve it. The troubleshooting process is shown in the following diagram:

To troubleshoot and resolve problems, you also need to have these two basic abilities: the ability to understand the content of error logs and find solutions based on error logs.

Let’s take an example. Suppose we have the following error:

[going@dev iam]$ mysql -h127.0.0.1 -uroot -p'iam59!z$'
bash: /usr/bin/mysql: No such file or directory
[going@dev iam]$

For this error, let’s first understand the error message: “mysql command not found” indicates that mysql is not installed or the installation of mysql has failed.

Therefore, our solution is to re-execute the steps to install MariaDB mentioned in Chapter 03:

$ cd $IAM_ROOT
$ ./scripts/install/mariadb.sh iam::mariadb::install

Next, I will use the iam-apiserver service as an example to show you how to troubleshoot and resolve problems specifically.

Identifying the Problem #

To troubleshoot, we need to identify the problem first. We usually use the following methods to identify the problem.

• Checking service status: After starting the iam-apiserver service, execute systemctl status iam-apiserver to check if the iam-apiserver service fails to start, that is, the Active value is not active (running).
• Functional abnormalities: Access the iam-apiserver service and find functional abnormalities or errors, such as returning values that differ from the expected results.
• Log errors: Find some WARN, ERROR, PANIC, FATAL, etc. level error logs in the iam-apiserver logs.

Locating the Problem #

After identifying the problem, we need to locate the root cause of the problem. We can use the following three methods to locate the problem.

• Viewing logs, which is the simplest troubleshooting method.
• Using the Go debugging tool Delve to locate the problem.
• Adding debug logs, tracing code from the entry point of the program, and printing debug logs in key locations to locate the problem.

In the process of locating the problem, we can adopt the strategy of “following the clues” to troubleshoot the problem. For example, our program execution flow is: A -> B -> … -> N. Where A, B, and N can all be considered as a debugging point. The so-called debugging point is the point where the problem needs to be located, and these points may be the root cause of the problem.

During the troubleshooting process, you can find the next debugging point B based on the error logs at the top level. If it is determined that there is no problem with B, then continue to find the next debugging point based on the program execution flow. Repeat this process until the final debugging point, which is the root cause N of the problem. N is the bug point. The execution flow is shown in the following diagram:

Now, let’s take a closer look at these three methods of locating the problem.

Locating the Problem by Viewing Logs #

We should first locate the problem through logs, as this is the simplest and most efficient way. To locate the problem through logs, you not only need to know how to view logs but also understand the reasons for the log errors.

Next, I will explain the steps to locate the problem using this method.

Step 1: Ensure that the service is running properly.

You can use the systemctl status command to check the running status of the service:

$ systemctl status iam-apiserver
● iam-apiserver.service - IAM APIServer
   Loaded: loaded (/etc/systemd/system/iam-apiserver.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-09-09 13:47:56 CST; 2s ago
     Docs: https://github.com/marmotedu/iam/blob/master/init/README.md
  Process: 119463 ExecStart=/opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml (code=exited, status=1/FAILURE)
  Process: 119461 ExecStartPre=/usr/bin/mkdir -p /var/log/iam (code=exited, status=0/SUCCESS)
  Process: 119460 ExecStartPre=/usr/bin/mkdir -p /data/iam/iam-apiserver (code=exited, status=0/SUCCESS)
 Main PID: 119463 (code=exited, status=1/FAILURE)

You can see that Active is not active (running), indicating that the iam-apiserver service is not running properly. From the information in the output, Process: 119463 ExecStart=/opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml (code=exited, status=1/FAILURE), we can obtain the following information:

• The startup command for the iam-apiserver service is /opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml.
• The configuration file loaded by /opt/iam/bin/iam-apiserver is /etc/iam/iam-apiserver.yaml.
• The command /opt/iam/bin/iam-apiserver failed to execute with an exit code of 1, and its process ID is 119463. Pay attention here, systemctl status will replace the second half of lines that exceed a certain length with ellipsis. If you want to view the complete information, you can append the -l parameter, that is, systemctl status -l, to view.

Since the iam-apiserver command failed to start, we need to check the log when iam-apiserver starts to see if there are any error logs.

Next, we will enter Step 2, checking the running log of iam-apiserver.

Let me mention here that if you are not familiar with systemd, you can take this opportunity to learn about it. You can refer to two blog posts by Ruan Yi-feng: Systemd Tutorial: Command Section and Systemd Tutorial: Practical Section.

So how do we check it? We have three ways to do it, and I have listed them below in order of priority. When you locate the problem and check the log, you can choose one of the three methods based on priority: 1 > 2 > 3.

1. Use journalctl -u iam-apiserver to view.
2. Use the iam-apiserver log file to view.
3. Use the console to view.

Now I will introduce these three viewing methods separately.

Let’s start with the highest priority method, using journalctl -u iam-apiserver to view.

Systemd provides its own logging system called the journal. We can use the journalctl command to read the journal logs. journalctl provides the -u option to view the logs of a specific unit, and provides _PID to view logs of a specified process ID. In Step 1, we know that the process ID of the failed service startup is 119463. Execute the following command to view the logs of this startup:

$ sudo journalctl _PID=119463
-- Logs begin at Thu 2021-09-09 09:12:25 CST, end at Thu 2021-09-09 14:40:48 CST. --
...
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: 2021-09-09 13:47:56.727        INFO        apiserver        [email protected]/gorm.go:202        mysql/mysql.go:75[error] faile>
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: 2021-09-09 13:47:56.727        FATAL        apiserver        apiserver/server.go:139        Failed to get cache instance: g>
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/marmotedu/iam/internal/apiserver.(*completedExtraConfig).New
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/server.go:139
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/marmotedu/iam/internal/apiserver.createAPIServer
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/server.go:66
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/marmotedu/iam/internal/apiserver.Run
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/run.go:11
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/marmotedu/iam/internal/apiserver.run.func1
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/app.go:46
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/marmotedu/iam/pkg/app.(*App).runCommand
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/pkg/app/app.go:278

Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/spf13/cobra.(*Command).execute
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:856
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/spf13/cobra.(*Command).ExecuteC
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:974
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/spf13/cobra.(*Command).Execute
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:902
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: github.com/marmotedu/iam/pkg/app.(*App).Run
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/pkg/app/app.go:233
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: main.main
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/workspace/golang/src/github.com/marmotedu/iam/cmd/iam-apiserver/apiserver.go:24
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]: runtime.main
Sep 09 13:47:56 VM-200-70-centos iam-apiserver[119463]:         /home/going/go/go1.16.2/src/runtime/proc.go:225
lines 10-54/54 (END)

From the above logs, we found the reason for the failure of the iam-apiserver startup: There was a FATAL level error occurring during the startup. At this point, you have already initially identified the cause of the problem.

Let’s also take a look at the method of viewing the logs through the iam-apiserver log file.

As an enterprise-level practical project, the logs of iam-apiserver are of course recorded in a log file. In the Step 1, we know from the information output by systemctl status iam-apiserver that the configuration file loaded during the iam-apiserver startup is /etc/iam/iam-apiserver.yaml. Therefore, we can check the location of the log file by looking at the log.output-paths configuration item in the iam-apiserver.yaml configuration file:

log:
    name: apiserver # Name of the logger
    development: true # Whether it is in development mode. If it is in development mode, a stack trace will be taken for the DPanicLevel logs.
    level: debug # Log level, from low to high priority: debug, info, warn, error, dpanic, panic, fatal.
    format: console # Supported log output format, currently supports console and json. Console is actually the text format.
    enable-color: true # Whether to enable color output, true: yes, false: no
    disable-caller: false # Whether to enable caller. If enabled, the file, function, and line number where the log is called will be displayed in the log.
    disable-stacktrace: false # Whether to disable printing stack trace information at PanicLevel and above
    output-paths: /var/log/iam/iam-apiserver.log,stdout # Supports output to multiple destinations, separated by commas. Supports output to standard output (stdout) and file.
    error-output-paths: /var/log/iam/iam-apiserver.error.log # Output paths for zap internal (non-business) error logs, multiple outputs separated by commas

As we can see, iam-apiserver logs are recorded in both /var/log/iam/iam-apiserver.log and stdout. Therefore, we can check the error message by viewing the iam-apiserver.log log file:

$ tail -25 /var/log/iam/iam-apiserver.log
...
2021-09-09 15:42:35.231	INFO	apiserver	server/genericapiserver.go:88	GET    /version --> github.com/marmotedu/iam/internal/pkg/server.(*GenericAPIServer).InstallAPIs.func2 (10 handlers)
2021-09-09 15:42:35.232	INFO	apiserver	[[email protected]](/cdn-cgi/l/email-protection)/gorm.go:202	mysql/mysql.go:75[error] failed to initialize database, got error dial tcp 127.0.0.1:3309: connect: connection refused
2021-09-09 15:42:35.232	FATAL	apiserver	apiserver/server.go:139	Failed to get cache instance: got nil cache server
github.com/marmotedu/iam/internal/apiserver.(*completedExtraConfig).New
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/server.go:139
github.com/marmotedu/iam/internal/apiserver.createAPIServer
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/server.go:66
github.com/marmotedu/iam/internal/apiserver.Run
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/run.go:11
github.com/marmotedu/iam/internal/apiserver.run.func1
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/app.go:46
github.com/marmotedu/iam/pkg/app.(*App).runCommand
	/home/going/workspace/golang/src/github.com/marmotedu/iam/pkg/app/app.go:278
github.com/spf13/cobra.(*Command).execute
	/home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	/home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:974
github.com/spf13/cobra.(*Command).Execute
	/home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:902
github.com/marmotedu/iam/pkg/app.(*App).Run
	/home/going/workspace/golang/src/github.com/marmotedu/iam/pkg/app/app.go:233
main.main
	/home/going/workspace/golang/src/github.com/marmotedu/iam/cmd/iam-apiserver/apiserver.go:24
runtime.main
	/home/going/go/go1.16.2/src/runtime/proc.go:225

Let’s take a look at the last way to view it, through the console.

Of course, we can also directly view the log through the console, which requires us to run iam-apiserver in the Linux terminal foreground (in the Step 1, we already know the startup command):

$ sudo /opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml
...
2021-09-09 15:47:00.660	INFO	apiserver	server/genericapiserver.go:88	GET    /debug/pprof/mutex --> github.com/gin-contrib/pprof.pprofHandler.func1 (10 handlers)
2021-09-09 15:47:00.660	INFO	apiserver	server/genericapiserver.go:88	GET    /debug/pprof/threadcreate --> github.com/gin-contrib/pprof.pprofHandler.func1 (10 handlers)
2021-09-09 15:47:00.660	INFO	apiserver	server/genericapiserver.go:88	GET    /version --> github.com/marmotedu/iam/internal/pkg/server.(*GenericAPIServer).InstallAPIs.func2 (10 handlers)
2021-09-09 15:47:00.661	INFO	apiserver	[[email protected]](/cdn-cgi/l/email-protection)/gorm.go:202	mysql/mysql.go:75[error] failed to initialize database, got error dial tcp 127.0.0.1:3309: connect: connection refused
2021-09-09 15:47:00.661	FATAL	apiserver	apiserver/server.go:139	Failed to get cache instance: got nil cache server
github.com/marmotedu/iam/internal/apiserver.(*completedExtraConfig).New
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/server.go:139
github.com/marmotedu/iam/internal/apiserver.createAPIServer
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/server.go:66
github.com/marmotedu/iam/internal/apiserver.Run
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/run.go:11
github.com/marmotedu/iam/internal/apiserver.run.func1
	/home/going/workspace/golang/src/github.com/marmotedu/iam/internal/apiserver/app.go:46
github.com/marmotedu/iam/pkg/app.(*App).runCommand
	/home/going/workspace/golang/src/github.com/marmotedu/iam/pkg/app/app.go:278
github.com/spf13/cobra.(*Command).execute
	/home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	/home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:974
github.com/spf13/cobra.(*Command).Execute
	/home/going/workspace/golang/pkg/mod/github.com/spf13/[[email protected]](/cdn-cgi/l/email-protection)/command.go:902
github.com/marmotedu/iam/pkg/app.(*App).Run
	/home/going/workspace/golang/src/github.com/marmotedu/iam/pkg/app/app.go:233
main.main
	/home/going/workspace/golang/src/github.com/marmotedu/iam/cmd/iam-apiserver/apiserver.go:24
runtime.main
	/home/going/go/go1.16.2/src/runtime/proc.go:225

Through the above three ways of viewing, we can preliminarily locate the cause of the service exception.

Use Go debugging tool Delve to locate the problem #

Viewing logs is the simplest troubleshooting method. By viewing the log, we can quickly solve the problem if we can locate the root cause of the problem. However, in some cases, we may not be able to locate the problem through the log, for example:

• The program is abnormal, but there are no error logs.
• The log has an error, but we can only judge the surface of the problem and cannot accurately find the root cause of the problem.

In these two cases, we need to further locate the problem. At this time, we can use the Delve debugging tool to try to locate the problem. You can refer to Delve User Guide for details on how to use Delve.

Add Debug logs to locate the problem #

If using the Delve tool still does not locate the problem, you can try the most primitive method next: add Debug logs to locate the problem. This method can be divided into two steps.

Step 1: Add Debug logs to critical code sections.

You need to decide on the critical code sections based on your understanding of the code. If you are not sure which code section is causing the problem, you can add Debug logs from the entry point of the request, and then follow the code flow step by step to troubleshoot, adding Debug logs where needed. For example, by checking the logs, we have identified that the code at internal/apiserver/server.go:139 is causing a FATAL error in the program. The FATAL error message is “Failed to get cache instance: got nil cache server”. The cache server is nil, indicating that the cache server was not initialized. Let’s take a look at the initialization function of the cache server:

func GetCacheInsOr(store store.Factory) (*Cache, error) {
    if store != nil {
        once.Do(func() {
            cacheServer = &Cache{store}
        })
    }

    if cacheServer == nil {
        return nil, fmt.Errorf("got nil cache server")
    }

    return cacheServer, nil
}

From this code, we can easily analyze that store == nil is causing the cacheServer to not be initialized. Let’s take a look at the initialization code for store and add some debug logs, as shown in the image:

After adding the debug code, we can recompile and run the program.

Here’s a little tip: you can add debug logs at the error return points to help you locate the error, for example:

if err != nil {
  log.Debugf("DEBUG POINT - 1: %v", err)
  return err
}

Step 2: Recompile the source code and start.

To make debugging and log viewing easier, we will directly run iam-apiserver in the Linux terminal as follows:

$ sudo /opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml

Let’s take a look at the printed content of the debug logs we added, as shown in the image:

From the debug logs, we can see that the port used to create the MySQL instance is incorrect. The correct port should be 3306, not 3309. The port for the MySQL server is configured in iam-apiserver.yaml. Modify iam-apiserver.yaml with the correct configuration and start it:

$ sudo /opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml

Let’s take a look at the console logs again, as shown in the image:

We can see that the issue has been fixed, dbIns is not nil, and the program is running normally:

$ systemctl status iam-apiserver
● iam-apiserver.service - IAM APIServer
   Loaded: loaded (/etc/systemd/system/iam-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-09-09 20:48:18 CST; 17s ago
     Docs: https://github.com/marmotedu/iam/blob/master/init/README.md
  Process: 255648 ExecStartPre=/usr/bin/mkdir -p /var/log/iam (code=exited, status=0/SUCCESS)
  Process: 255647 ExecStartPre=/usr/bin/mkdir -p /data/iam/iam-apiserver (code=exited, status=0/SUCCESS)
 Main PID: 255650 (iam-apiserver)
    Tasks: 5 (limit: 23724)
   Memory: 7.3M
   CGroup: /system.slice/iam-apiserver.service
           └─255650 /opt/iam/bin/iam-apiserver --config=/etc/iam/iam-apiserver.yaml

Here, Active is in the active (running) state.

Since these debug logs can assist you in troubleshooting and help you locate problems, it indicates that these logs are useful, so you can keep these debug log calls.

Problem Resolution #

During the troubleshooting phase, we have identified the cause of the problem. Now, based on your understanding of the business and underlying code implementation, you can proceed to fix the problem. There is no unified process or methodology on how to fix it, as it depends on the specific situation. I won’t go into too much detail on how to fix it here.

Above, I have explained the thought process and methods for troubleshooting. Next, I will show you 9 common problems you may encounter when deploying and using the IAM system, and provide solutions. These problems are mostly caused by server environment issues.

Common Issues and Solutions for IAM #

Issue 1: Error No match for argument: neovim when installing neovim.

The solution is to install the EPEL repository:

$ sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

Issue 2: Failure to install protoc-gen-go (timeout, error, etc.).

This issue may occur because the network environment of your server is unable to access github.com, or the access to github.com is too slow.

The solution is to manually compile and install it with the following steps:

$ git clone --depth 1 https://github.com/golang/protobuf $GOPATH/src/github.com/golang/protobuf
$ cd $GOPATH/src/github.com/golang/protobuf/protoc-gen-go
$ go install -v .

Issue 3: Encounter errors like xxx: permission denied.

This kind of error occurs because you do not have the permission to execute the current operation. The solution is to check whether you have the permission to execute the current operation. If you do not have the permission, you need to switch to a user with the required permission or give up executing the current operation.

To explain the issue, here is an example of an error and a troubleshooting approach. The error log is as follows:

[going@VM-8-9-centos /]$ go get -u github.com/golang/protobuf/protoc-gen-go
go: could not create module cache: mkdir /golang: permission denied
[going@VM-8-9-centos /]$ sudo go get -u github.com/golang/protobuf/protoc-gen-go
sudo: go: command not found

In the above error, two errors are reported: mkdir /golang: permission denied and sudo: go: command not found. Let’s first look at the error mkdir /golang: permission denied.

From the command prompt $, we can see that the current logged-in user is a regular user. From the error message mkdir /golang: permission denied, we can see that the go get -u github.com/golang/protobuf/protoc-gen-go command executed mkdir /golang underneath. Since the regular user does not have write permission for the / directory, a permission error is reported. The solution is to switch to the user’s directory and execute the go get -u command.

Now let’s look at the error sudo: go: command not found. The sudo command switches the command execution environment to the root user. Obviously, the root user does not have the go command installed, causing the command not found error. The solution is to remove sudo and directly execute $ go get -u xxx.

Issue 4: Various errors when using VimIDE.

The cause of these errors is related to the environment, system environment during the installation of VimIDE, package versions, etc. Since there are too many types of errors, it is not possible to explain them all. Therefore, I suggest you ignore these errors as they do not affect later learning.

Issue 5: Error {"code":100202,"message":"Signature is invalid"} when accessing the /v1/authz interface of iam-authz-server.

This may be due to problems with the issued token. It is recommended to re-execute the following 5 steps:

1. Login to the system again and obtain the access token:

$ token=`curl -s -XPOST -H'Content-Type: application/json' -d'{"username":"admin","password":"Admin@2021"}' http://127.0.0.1:8080/login | jq -r .token`

If the jq command is not installed, you can install it by running the command sudo yum -y install jq.

2. Create an authorization policy:

$ curl -s -XPOST -H"Content-Type: application/json" -H"Authorization: Bearer $token" -d'{"metadata":{"name":"authztest"},"policy":{"description":"One policy to rule them all.","subjects":["users:<peter|ken>","users:maria","groups:admins"],"actions":["delete","<create|update>"],"effect":"allow","resources":["resources:articles:<.*>","resources:printer"],"conditions":{"remoteIPAddress":{"type":"CIDRCondition","options":{"cidr":"192.168.0.1/16"}}}}}' http://127.0.0.1:8080/v1/policies

3. Create a secret and extract the secretID and secretKey from the command output:

$ curl -s -XPOST -H"Content-Type: application/json" -H"Authorization: Bearer $token" -d'{"metadata":{"name":"authztest"},"expires":0,"description":"admin secret"}' http://127.0.0.1:8080/v1/secrets
{"metadata":{"id":23,"name":"authztest","createdAt":"2021-04-08T07:24:50.071671422+08:00","updatedAt":"2021-04-08T07:24:50.071671422+08:00"},"username":"admin","secretID":"ZuxvXNfG08BdEMqkTaP41L2DLArlE6Jpqoox","secretKey":"7Sfa5EfAPIwcTLGCfSvqLf0zZGCjF3l8","expires":0,"description":"admin secret"}

1. Generate Token to access iam-authz-server

iamctl provides the jwt sign command which allows you to issue a Token based on the secretID and secretKey for convenient usage. The specific command to issue a Token is as follows:

$ iamctl jwt sign ZuxvXNfG08BdEMqkTaP41L2DLArlE6Jpqoox 7Sfa5EfAPIwcTLGCfSvqLf0zZGCjF3l8 # iamctl jwt sign $secretID $secretKey
eyJhbGciOiJIUzI1NiIsImtpZCI6Ilp1eHZYTmZHMDhCZEVNcWtUYVA0MUwyRExBcmxFNkpwcW9veCIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJpYW0uYXV0aHoubWFybW90ZWR1LmNvbSIsImV4cCI6MTYxNzg0NTE5NSwiaWF0IjoxNjE3ODM3OTk1LCJpc3MiOiJpYW1jdGwiLCJuYmYiOjE2MTc4Mzc5OTV9.za9yLM7lHVabPAlVQLCqXEaf8sTU6sodAsMXnmpXjMQ

2. Test resource authorization:

$ curl -s -XPOST -H'Content-Type: application/json' -H'Authorization: Bearer eyJhbGciOiJIUzI1NiIsImtpZCI6Ilp1eHZYTmZHMDhCZEVNcWtUYVA0MUwyRExBcmxFNkpwcW9veCIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJpYW0uYXV0aHoubWFybW90ZWR1LmNvbSIsImV4cCI6MTYxNzg0NTE5NSwiaWF0IjoxNjE3ODM3OTk1LCJpc3MiOiJpYW1jdGwiLCJuYmYiOjE2MTc4Mzc5OTV9.za9yLM7lHVabPAlVQLCqXEaf8sTU6sodAsMXnmpXjMQ' -d'{"subject":"users:maria","action":"delete","resource":"resources:articles:ladon-introduction","context":{"remoteIPAddress":"192.168.0.5"}}' http://127.0.0.1:9090/v1/authz
{"allowed":true}

Question 6: Executing iamctl user list returns error: {"code":100207,"message":"Permission denied"}.

In this case, it may be due to incorrect password configuration.

Please check if the username and password in the $HOME/.iam/iamctl.yaml configuration file are set to admin, and if the password for admin is Admin@2021.

Question 7: When creating a user, an error {"code":100101,"message":"Database error"} is returned.

In this case, it may be due to a duplicate username. It is recommended to use a different username and try creating the user again.

Question 8: Errors such as No such file or directory, command not found, permission denied are reported.

For such errors, you need to troubleshoot and resolve the problem based on the error messages.

• No such file or directory: Confirm if the file exists, and check the reason for its nonexistence.
• command not found: Confirm if the command exists, if not, you can reinstall the command.
• permission denied: Confirm whether you have the necessary permissions. If not, switch to a user or directory with the appropriate permissions.

Question 9: The files iam-apiserver.service, /opt/iam/bin/iam-apiserver, and /etc/iam/iam-apiserver.yaml are reported as not found.

Let me explain the purpose of these files.

• /etc/systemd/system/iam-apiserver.service: The systemd Unit file for iam-apiserver.
• /opt/iam/bin/iam-apiserver: The binary startup command for iam-apiserver.
• /etc/iam/iam-apiserver.yaml: The configuration file for iam-apiserver.

If any of these files are missing, you will need to reinstall them. Let me provide the installation methods for these three files.

Installation method for /etc/systemd/system/iam-apiserver.service:

$ cd $IAM_ROOT
$ ./scripts/genconfig.sh scripts/install/environment.sh init/iam-apiserver.service > iam-apiserver.service
$ sudo mv iam-apiserver.service /etc/systemd/system/

Installation method for /opt/iam/bin/iam-apiserver:

$ cd $IAM_ROOT
$ source scripts/install/environment.sh
$ make build BINS=iam-apiserver
$ sudo cp _output/platforms/linux/amd64/iam-apiserver ${IAM_INSTALL_DIR}/bin

Installation method for /etc/iam/iam-apiserver.yaml:

$ cd $IAM_ROOT
$ ./scripts/genconfig.sh scripts/install/environment.sh configs/iam-apiserver.yaml > iam-apiserver.yaml
$ sudo mv iam-apiserver.yaml ${IAM_CONFIG_DIR}

Summary #

In this lecture, I used the iam-apiserver service as an example to introduce you to the basic troubleshooting process: discovering problems -> locating problems -> solving problems.

You can discover problems in three ways.

• Checking Service Status: After starting the iam-apiserver service, execute systemctl status iam-apiserver to discover if the iam-apiserver failed to start, indicated by the Active value not being active (running).
• Functional Abnormalities: Access the iam-apiserver service and observe functional abnormalities or errors, such as unexpected return values or interface errors.
• Log Errors: Discover high-level error logs such as WARN, ERROR, PANIC, FATAL in the iam-apiserver logs.

After discovering a problem, you can use three methods to locate the problem: viewing logs, using the Go debugging tool Delve, and adding debug logs.

• Viewing Logs: Viewing logs is the simplest troubleshooting method.
• Using Delve, a Go debugging tool, to locate the problem.
• Adding Debug Logs: Trace the code from the program entry point and add debug logs at key locations to locate the problem.

Once you have identified the root cause of the problem, you need to solve it. You should solve the problem based on your understanding and mastery of the business and underlying code implementation.

Finally, I showcased nine common problems that you may encounter when deploying and using the IAM system, and provided solutions to help you.

After-class exercise #

1. Think about how to find the path of the systemd Unit file for iam-apiserver?
2. Execute the following commands:

$ token=`curl -s -XPOST -H'Content-Type: application/json' -d'{"username":"admin","password":"Admin@2021"}' http://127.0.0.1:8080/login | jq -r .token`
$ echo $token

You can obtain the token, but you found that the value of token is empty. Please provide your troubleshooting process and methods.

Feel free to discuss and communicate with me in the comment section. See you in the next lecture.