35 Efficiency Tools How to Design and Implement a Command Line Client Tool

35 Efficiency Tools How to Design and Implement a Command-Line Client Tool #

Hello, I’m Kong Lingfei. Today, let’s talk about how to implement a command line client tool.

If you’ve used Kubernetes, Istio, or etcd, then you must have used the command line tools provided by these open-source projects: kubectl, istioctl, etcdctl. It has become a trend for an xxx project to come with an xxxctl command line tool, and this is particularly common in large systems. Providing an xxxctl command line tool has two advantages:

  • Automation: The xxxctl tool can be called in scripts to achieve automation.
  • Efficiency: By encapsulating the functionalities of an application into commands and parameters, it becomes convenient for administrators and developers to invoke them on Linux servers.

Among them, kubectl has the most complex and outstanding design. The iamctl client tool of the IAM project is implemented following the example of kubectl. In this talk, I will dissect the implementation of the iamctl command line tool to introduce how to create an excellent client tool.

Introduction to Common Clients #

Before introducing the implementation of the iamctl command line tool, let’s take a look at common clients.

Clients, also known as user clients, are installed on the client machine and are used by users to access backend services. Different clients target different audiences and provide different access capabilities. The common clients include the following:

  • Front-end clients, including browsers and mobile applications;
  • SDKs;
  • Command line tools;
  • Other terminals.

Next, let me introduce each of them separately.

Browsers and mobile applications provide an interactive interface for users to access backend services. They have the best user experience and are aimed at end users. These two types of clients are also called front-end clients. Front-end clients are developed by front-end developers and invoke backend services through API interfaces. Backend developers do not need to worry about these two types of clients, they only need to focus on how to provide API interfaces.

SDK (Software Development Kit) is also a client for developers to call. When developers call APIs, if they use the HTTP protocol, they need to write HTTP invocation code, encapsulate HTTP request packages and unpack response packages, and handle HTTP status codes, which is not very convenient. The SDK actually encapsulates a series of functions of the API interface. Developers can call the API interface by calling the functions in the SDK. The main purpose of the SDK is to facilitate developers to call the API interface and reduce their workload.

Command line tools are executable binary programs that can be executed on an operating system. They provide a more convenient and efficient way to access backend services than SDKs and API interfaces. They are used directly by operations or developers on servers or called in automation scripts.

There are also various other clients, and I will list some common ones here.

  • Terminal devices: POS machines, learning machines, smart speakers, etc.
  • Third-party applications: By calling API interfaces or SDKs, they can use the backend services we provide to implement their own functions.
  • Scripts: In scripts, backend services can be called through API interfaces or command line tools to achieve automation.

These various other clients all use backend services through API interfaces, just like front-end clients, and do not require backend developers to develop.

SDKs and command line tools are the clients that require backend developers to invest resources in development. These two types of client tools have a calling and being called sequence, as shown in the diagram below:

Image

As you can see, both the command line tool and the SDK ultimately call the backend services through the API interface. This approach ensures consistency of the services and reduces the additional development workload required to adapt to multiple clients.

Characteristics of the Client for Large-scale Systems (xxxctl) #

By studying excellent command-line tools such as kubectl, istioctl, and etcdctl, we can observe that a command-line tool for a large-scale system typically has the following characteristics:

  • Supports commands and subcommands, with their own unique command-line parameters.
  • Supports some special commands. For example, it supports the completion command, which can output bash/zsh autocomplete scripts to achieve command and parameter autocompletion. It also supports the version command, which can not only output the version of the client but also the version of the server (if necessary).
  • Supports global options, which can be used as command-line parameters for all commands and subcommands.
  • Supports -h/help, which can print the help information for xxxctl. For example:
$ iamctl -h
iamctl controls the iam platform, is the client side tool for iam platform.

Find more information at:
https://github.com/marmotedu/iam/blob/master/docs/guide/en-US/cmd/iamctl/iamctl.md

Basic Commands:
  info        Print the host information
  color       Print colors supported by the current terminal
  new         Generate demo command code
  jwt         JWT command-line tool

Identity and Access Management Commands:
  user        Manage users on iam platform
  secret      Manage secrets on iam platform
  policy      Manage authorization policies on iam platform

Troubleshooting and Debugging Commands:
  validate    Validate the basic environment for iamctl to run

Settings Commands:
  set         Set specific features on objects
  completion  Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  version     Print the client and server version information

Usage:
  iamctl [flags] [options]

Use "iamctl <command> --help" for more information about a given command.
Use "iamctl options" for a list of global command-line options (applies to all commands).
  • Supports xxxctl help [command | command subcommand] [command | command subcommand] -h, which prints the help information of a command/subcommand in the format of command description + usage. For example:
$ istioctl help register
Registers a service instance (e.g. VM) joining the mesh

Usage:
  istioctl register <svcname> <ip> [name1:]port1 [name2:]port2 ... [flags]

In addition, a command-line tool for a large-scale system can also support more advanced functions, such as command grouping, configuration files, usage examples for commands, etc.

In the Go ecosystem, if we are looking for a command-line tool that meets all of the above characteristics, nothing can beat kubectl. Because the client tool iamctl that I am going to talk about today is implemented based on it, I won’t go into detail about kubectl here. However, I still recommend you to study kubectl’s implementation carefully.

Core Implementation of iamctl #

Next, let me introduce iamctl, the client tool that comes with IAM system. It is implemented based on kubectl and meets the requirements of a large-scale system client tool. I will cover the functionality, code structure, command line options, and configuration file parsing of iamctl.

Functionality of iamctl #

iamctl classifies commands. I suggest you also classify commands because it not only helps you understand the purpose of commands but also allows you to quickly locate a certain type of command. Additionally, when there are many commands, classification can make the commands appear more organized.

The implemented commands of iamctl are as follows:

Image

For more detailed functionality, you can refer to iamctl -h. I recommend considering implementing the following features when implementing the xxxctl tool:

  • API functionality: All API functionalities of the platform can be conveniently accessed through xxxctl.
  • Utilities: Some useful functions when using the IAM system, such as issuing JWT tokens.
  • version, completion, and validate commands.

Code Structure #

The main function of the iamctl tool is located in the file iamctl.go. The implementation of commands is stored in the file internal/iamctl/cmd/cmd.go. The commands of iamctl are uniformly stored in the directory internal/iamctl/cmd, where each command is a Go package with the package name being the command name, and the specific implementation is stored in the file internal/iamctl/cmd/<command>/<command>.go. If a command has subcommands, the implementation of the subcommands is stored in the file internal/iamctl/cmd/<command>/<command>_<subcommand>.go.

Using this code organization method can make the code orderly, making it easy to locate and maintain the code even when there are many commands.

Command Line Options #

The code for adding command line options is in the function NewIAMCtlCommand, and the core code snippet is as follows:

flags := cmds.PersistentFlags()
...
iamConfigFlags := genericclioptions.NewConfigFlags(true).WithDeprecatedPasswordFlag().WithDeprecatedSecretFlag()
iamConfigFlags.AddFlags(flags)
matchVersionIAMConfigFlags := cmdutil.NewMatchVersionFlags(iamConfigFlags)
matchVersionIAMConfigFlags.AddFlags(cmds.PersistentFlags())

NewConfigFlags(true) returns parameters with default values and adds them to the command line flags of cobra through iamConfigFlags.AddFlags(flags).

The return values of NewConfigFlags(true) are all pointers to struct types. The advantage of doing this is that the program can determine whether a certain parameter is specified, so parameters can be added as needed. For example, password and secret authentication parameters can be added using WithDeprecatedPasswordFlag() and WithDeprecatedSecretFlag().

NewMatchVersionFlags specifies whether the server version and client version need to be consistent. If they are inconsistent, an error will be reported when calling the service interface.

Configuration File Parsing #

iamctl needs to connect to the iam-apiserver to perform user, policy, and key CRUD operations, and it needs to be authenticated. To achieve these functions, there are many configuration options. It would be cumbersome and error-prone to specify these options every time on the command line.

The best way is to save them in a configuration file and load the configuration file. The code for loading the configuration file is in the function NewIAMCtlCommand, the code snippet is as follows:

_ = viper.BindPFlags(cmds.PersistentFlags())
cobra.OnInitialize(func() {
    genericapiserver.LoadConfig(viper.GetString(genericclioptions.FlagIAMConfig), "iamctl")
})

iamctl loads the configuration file based on the following priority:

  1. The configuration file specified by the command line parameter --iamconfig.
  2. The iamctl.yaml file in the current directory.
  3. The $HOME/.iam/iamctl.yaml file.

This way of loading the configuration file has two advantages. Firstly, different configuration files can be manually specified, which is particularly important in multi-environment and multi-configuration scenarios. Secondly, it is convenient to use because the configuration can be stored in the default loading path, and the --iamconfig parameter does not need to be specified when executing commands.

Once the configuration file is loaded, the configurations can be obtained using the viper.Get<Type>() function. For example, iamctl uses the following viper.Get<Type> methods:

Image

How are subcommands constructed in iamctl? #

After discussing the core implementation of the iamctl command-line tool, let’s take a look at how subcommands are constructed in iamctl.

The core capability of a command-line tool is to provide various commands to perform different functions. Each command can be constructed in different ways, but it is best to use a consistent method and abstract it into a model. As shown in the diagram below:

Image

You can group the commands provided by a command-line tool. Each group contains multiple commands, and each command can have multiple subcommands. The construction method for subcommands is identical to that of the parent commands.

Each command can be constructed in one of the following four ways. You can refer to the specific code in internal/iamctl/cmd/user/user_update.go for details.

  • Create the command framework using the NewCmdXyz function. NewCmdXyz function creates a cobra.Command variable to create the command. The cobra.Command struct type’s Short, Long, and Example fields are specified to define the command’s usage documentation (iamctl -h), detailed usage documentation (iamctl xyz -h), and examples of usage.
  • Add command line options to the command using cmd.Flags().XxxxVar.
  • To allow the command to be executed in the default way when no command line arguments are specified, return a XyzOptions type variable that has default options using the NewXyzOptions function.
  • The XyzOptions option has three methods: Complete, Validate, and Run, which respectively handle option completion, option validation, and command execution. The command’s execution logic can be written in the func (o *XyzOptions) Run(args []string) error function.

By constructing commands in the same way and abstracting them into a common model, there are four benefits:

  • Reduced learning curve: Understanding the construction method of one command allows understanding of the construction method of other commands.
  • Improved development efficiency of new commands: Development frameworks of existing commands can be reused, and new commands only need to fill in the business logic.
  • Automatic command generation: New commands can be generated automatically according to a predefined command model.
  • Easy maintenance: Because all commands are derived from the same command model, consistent code style can be maintained, making it easier for future maintenance.

Automatic command generation #

As mentioned earlier, one of the benefits of generating a command model automatically is the ability to generate commands automatically. Let’s take a closer look at the specific steps below.

iamctl comes with a command generation tool. The generation process can be divided into 5 steps. Here, we assume that we want to generate the xyz command.

Step 1: Create a xyz directory to store the source code of the xyz command:

$ mkdir internal/iamctl/cmd/xyz

Step 2: Generate the source code of the xyz command using the iamctl new command in the xyz directory:

$ cd internal/iamctl/cmd/xyz/
$ iamctl new xyz
Command file generated: xyz.go

Step 3: Add the xyz command to the root command. Assuming xyz belongs to the Settings Commands command group.

In the NewIAMCtlCommand function, find the Settings Commands group and append NewCmdXyz to the Commands array:

       {
            Message: "Settings Commands:",
            Commands: []*cobra.Command{
                set.NewCmdSet(f, ioStreams),
                completion.NewCmdCompletion(ioStreams.Out, ""),
                xyz.NewCmdXyz(f, ioStreams),
            },
        },

Step 4: Compile iamctl:

$ make build BINS=iamctl

Step 5: Test:

$ iamctl xyz -h

A longer description that spans multiple lines and likely contains examples and usage of using your command. For
example:
 
 Cobra is a CLI library for Go that empowers applications. This application is a tool to generate the needed files to
quickly create a Cobra application.
 
Examples:
  # Print all option values for xyz
  iamctl xyz marmotedu marmotedupass
 
Options:
  -b, --bool=false: Bool option.
  -i, --int=0: Int option.
      --slice=[]: String slice option.
      --string='default': String option.
 
Usage:
  iamctl xyz USERNAME PASSWORD [options]
 
Use "iamctl options" for a list of global command-line options (applies to all commands).
$ iamctl xyz marmotedu marmotedupass
The following is option values:
==> --string: default(complete)
==> --slice: []
==> --int: 0
==> --bool: false
 
The following is args values:
==> username: marmotedu
==> password: marmotedupass

You can see that with just a few steps, a new command xyz has been added. The iamctl new command can not only generate commands without subcommands, but also commands with subcommands, as shown below:

$ iamctl new -g xyz
Command file generated: xyz.go
Command file generated: xyz_subcmd1.go
Command file generated: xyz_subcmd2.go

Command Autocompletion #

Cobra automatically generates autocompletion scripts based on registered commands, allowing autocompletion for parent commands, subcommands, and options. In bash, you can configure autocompletion as follows.

Step 1: Generate the autocompletion script:

$ iamctl completion bash > ~/.iam/completion.bash.inc

Step 2: Load bash with the autocompletion script upon login:

$ echo "source '$HOME/.iam/completion.bash.inc'" >> $HOME/.bash_profile
$ source $HOME/.bash_profile

Step 3: Test the autocompletion feature:

$ iamctl xy<TAB> # Press TAB, autocompletes to: iamctl xyz
$ iamctl xyz --b<TAB> # Press TAB, autocompletes to: iamctl xyz --bool

Enhanced Output #

When developing commands, you can improve the user experience by using some techniques. I often print colored outputs or output tables, as shown in the image below:

Image

Here, I use the github.com/olekukonko/tablewriter package to create tables and the github.com/fatih/color package to print colored strings. For specific usage examples, you can refer to the internal/iamctl/cmd/validate/validate.go file.

The github.com/fatih/color package allows you to colorize strings. The correspondence between strings and their colors can be viewed using iamctl color, as shown in the image below:

Image

How does iamctl make API calls? #

In the previous section, I introduced the construction method of the iamctl command. Now let’s take a look at how iamctl makes API calls to the server.

The functionality of a Go backend service is usually exposed through API interfaces. A backend service may be used by many clients, such as browsers, command line tools, and mobile phones. To maintain consistency of functionality, these clients all call the same set of APIs to complete the same tasks, as shown in the following diagram:

Image

If a command line tool needs to use the functionality of the backend service, it also needs to make API calls. Ideally, all the API functionalities exposed by the Go backend service can be completed through the command line tool. An API interface corresponds to a command, where the parameters of the API interface map to the parameters of the command.

To make API calls to the server, the most convenient method is to use the SDK. For some interfaces that do not have an SDK implementation, they can also be called directly. Therefore, the command line tool needs to support the following two types of call methods:

  • Making API calls to the server through the SDK.
  • Making direct API calls to the server (this column is about REST API interfaces).

iamctl creates a variable of type Factory through cmdutil.NewFactory, where Factory is defined as:

type Factory interface {
    genericclioptions.RESTClientGetter
    IAMClientSet() (*marmotedu.Clientset, error)
    RESTClient() (*restclient.RESTClient, error)
}

The variable f is passed into the command, and in the command, the RESTClient() and IAMClientSet() methods provided by the Factory interface are used to return the RESTful API client and the SDK client, respectively. The client interfaces are then used to make API calls. You can refer to the code in internal/iamctl/cmd/version/version.go for details.

Client Configuration File #

To create the RESTful API client and the SDK client, you need to invoke the f.ToRESTConfig() function to return a configuration variable of type *github.com/marmotedu/marmotedu-sdk-go/rest.Config, and then create clients based on the rest.Config type configuration variable.

The f.ToRESTConfig() function ultimately calls the toRawIAMConfigLoader function to generate the configuration. Here is the code:

func (f *ConfigFlags) toRawIAMConfigLoader() clientcmd.ClientConfig {
    config := clientcmd.NewConfig()
    if err := viper.Unmarshal(&config); err != nil {
        panic(err)
    }

    return clientcmd.NewClientConfigFromConfig(config)
}

The toRawIAMConfigLoader function returns a variable of type clientcmd.ClientConfig. The clientcmd.ClientConfig type provides a ClientConfig method, which returns a *rest.Config type variable.

Inside the toRawIAMConfigLoader function, the configuration stored in viper is parsed into a variable of type clientcmd.Config struct by using viper.Unmarshal. The configuration stored in viper is loaded when the cobra command is started through the LoadConfig function, as shown below (in the NewIAMCtlCommand function):

cobra.OnInitialize(func() {                  
    genericapiserver.LoadConfig(viper.GetString(genericclioptions.FlagIAMConfig), "config")
})

You can specify the path to the configuration file using the --config option.

SDK Invocation #

The SDK client is returned through IAMClient, as shown in the following code:

func (f *factoryImpl) IAMClient() (*iam.IamClient, error) {
    clientConfig, err := f.ToRESTConfig()
    if err != nil {
        return nil, err
    }
    return iam.NewForConfig(clientConfig)
}

marmotedu.Clientset provides all the interfaces of the iam-apiserver.

REST API Invocation #

The RESTful API client is returned through RESTClient(), as shown in the following code:

func (f *factoryImpl) RESTClient() (*restclient.RESTClient, error) {
    clientConfig, err := f.ToRESTConfig()
    if err != nil {
        return nil, err
    }
    setIAMDefaults(clientConfig)
    return restclient.RESTClientFor(clientConfig)
}

You can access the RESTful API interface in the following way:

serverVersion *version.Info

client, _ := f.RESTClient()
if err := client.Get().AbsPath("/version").Do(context.TODO()).Into(&serverVersion); err != nil {
    return err
}

The above code requests the /version endpoint of the iam-apiserver and saves the result in the serverVersion variable.

Summary #

In this lecture, I mainly analyzed the implementation of the iamctl command-line tool and then introduced how to implement an excellent client tool.

For a large system like xxx, there is usually a xxxctl command-line tool that can facilitate the development and operation of the system and achieve automation.

The IAM project, inspired by kubectl, has implemented the command-line tool iamctl. iamctl integrates many functionalities, and we can use these functionalities through iamctl subcommands. For example, we can use iamctl to perform CRUD operations on users, keys, and policies; we can set up auto-completion scripts for iamctl; we can also view the version information of the IAM system. Furthermore, you can even use the iamctl new command to quickly create a template for a iamctl subcommand.

iamctl is built using the cobra, pflag, and viper packages. Each subcommand contains some basic functionalities such as a short description, a long description, usage examples, command-line options, and option validation. iamctl can load different configuration files to connect to different clients. iamctl invokes server-side API interfaces using both SDK calls and REST API calls.

Practice Exercises #

  1. Try adding a cliprint subcommand to iamctl that reads and prints command line options.
  2. Think about other good ways to build command line tools. Feel free to share in the comments section.

Feel free to engage in discussion and exchange ideas in the comments section. See you in the next lesson.