19 Error Handling How to Design Error Wrappers #

Hello, I’m Kong Lingfei.

In Go project development, handling errors is essential. In addition to error codes we learned in the previous lesson, error handling also involves the use of error packages.

There are many excellent open-source error packages available in the industry, such as the “errors” package included in the Go standard library and the “github.com/pkg/errors” package. However, these packages currently do not support business error codes, making it difficult to meet the requirements of production-level applications. Therefore, in practical development, it is necessary to develop an error package that suits our own error code design. Of course, we don’t need to start from scratch, but can build upon some excellent existing packages.

In this lesson, let’s take a look at how to design an error package that adapts to the error codes we designed in the previous lesson, along with the implementation of a specific error code.

What features should an error package have? #

To design an excellent error package, we first need to know what features it should have. In my opinion, it should have at least the following six features:

Firstly, it should support error stack traces. Let’s take a look at the following code, which is assumed to be saved in the bad.go file:

package main

import (
    "fmt"
    "log"
)

func main() {
    if err := funcA(); err != nil {
        log.Fatalf("call func got failed: %v", err)
        return
    }

    log.Println("call func success")
}

func funcA() error {
    if err := funcB(); err != nil {
        return err
    }

    return fmt.Errorf("func called error")
}

func funcB() error {
    return fmt.Errorf("func called error")
}

When executing the code above, we get the following output:

$ go run bad.go
2021/07/02 08:06:55 call func got failed: func called error
exit status 1

At this point, we want to locate the problem, but we don’t know which line of code is causing the error, so we can only guess, and we may not even guess correctly. To solve this problem, we can add some debug information to help us locate the problem. This is not a problem in the test environment, but in the production environment, it is difficult to modify and release, and the problem may be difficult to reproduce. At this point, we might think that it would be great if we could print an error stack trace. For example:

2021/07/02 14:17:03 call func got failed: func called error
main.funcB
    /home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/good.go:27
main.funcA
    /home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/good.go:19
main.main
    /home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/good.go:10
runtime.main
    /home/colin/go/go1.16.2/src/runtime/proc.go:225
runtime.goexit
    /home/colin/go/go1.16.2/src/runtime/asm_amd64.s:1371
exit status 1

Based on the error output above, we can easily know which line of code is causing the error, greatly improving the efficiency of problem locating and reducing the difficulty of locating. Therefore, in my opinion, a good errors package should first support error stack traces.

Secondly, it should support different printing formats. For example, %+v, %v, %s, etc., can print error information with different levels of detail according to needs.

Thirdly, it should support the Wrap/Unwrap feature, which means appending new information to the existing error. For example, errors.Wrap(err, "open file failed"). The Wrap function is usually used in the calling function, and the calling function can wrap some of its own information based on the error reported by the called function to enrich the error information and facilitate later error locating. For example:

func funcA() error {
    if err := funcB(); err != nil {
        return errors.Wrap(err, "call funcB failed")
    }

    return errors.New("func called error")
}

func funcB() error {
    return errors.New("func called error")
}

It should be noted that the logic of the Wrap function can be different for different error types. In addition, when calling Wrap, an error stack trace node will also be generated. Since we can nest errors, sometimes we may also need to get the nested error. In this case, the error package needs to provide the Unwrap function.

Moreover, the error package should have the Is method.

In actual development, we often need to determine whether an error is a specific error. In Go 1.13 and earlier, when there was no wrapping error, we could use the following method to determine whether an error is the same:

if err == os.ErrNotExist {
    // normal code
}

But now, because of wrapping error, this kind of judgment will not work. You don’t even know if the returned err is a nested error or how many layers it has. In this case, our error package needs to provide the Is function:

func Is(err, target error) bool

If err and target are the same, or if err is a wrapping error and target is included in this nested error chain, it returns true; otherwise, it returns false.

In addition, the error package should support the As function.

In Go 1.13 and earlier, when there was no wrapping error, to convert an error to another error, we generally used type assertion or type switch, namely type assertion. For example:

if perr, ok := err.(*os.PathError); ok {
    fmt.Println(perr.Path)
}

But now, the returned err may be a nested error, even with several layers of nesting, so this method cannot be used. So, we can use the As function to implement this functionality. Now, let’s implement the above example using the As function:

var perr *os.PathError
if errors.As(err, &perr) {
    fmt.Println(perr.Path)
}

This way, we can fully implement the functionality of type assertion, and it is even more powerful because it can handle wrapping errors.

Finally, it should support two ways of creating errors: non-formatted creation and formatted creation. For example:

errors.New("file not found")
errors.Errorf("file %s not found", "iam-apiserver")

Above, we have introduced the features that an excellent error package should have. The good news is that there are many error packages on GitHub that implement these features, with github.com/pkg/errors being the most popular. Therefore, based on the github.com/pkg/errors package, I have made secondary encapsulation to support the error codes described in the previous lesson.

Error Package Implementation #

After identifying the features that a good error package should have, let’s take a look at the implementation of the error package. The source code is stored in github.com/marmotedu/errors.

I introduced a new withCode struct in the file github.com/pkg/errors/errors.go to introduce a new type of error that can record error codes, stack traces, causes, and specific error messages.

type withCode struct {
    err   error // underlying error
    code  int // error code
    cause error // cause error
    *stack // error stack trace
}

Now, let’s take a look at the functionality provided by github.com/marmotedu/errors through an example. Suppose the following code is saved in the errors.go file:

package main

import (
	"fmt"

	"github.com/marmotedu/errors"
	code "github.com/marmotedu/sample-code"
)

func main() {
    if err := bindUser(); err != nil {
        // %s: Returns the user-safe error string mapped to the error code or the error message if none is specified.
        fmt.Println("====================> %s <====================")
        fmt.Printf("%s\n\n", err)

        // %v: Alias for %s.
        fmt.Println("====================> %v <====================")
        fmt.Printf("%v\n\n", err)

        // %-v: Output caller details, useful for troubleshooting.
        fmt.Println("====================> %-v <====================")
        fmt.Printf("%-v\n\n", err)

        // %+v: Output full error stack details, useful for debugging.
        fmt.Println("====================> %+v <====================")
        fmt.Printf("%+v\n\n", err)

        // %#-v: Output caller details, useful for troubleshooting with JSON formatted output.
        fmt.Println("====================> %#-v <====================")
        fmt.Printf("%#-v\n\n", err)

        // %#+v: Output full error stack details, useful for debugging with JSON formatted output.
        fmt.Println("====================> %#+v <====================")
        fmt.Printf("%#+v\n\n", err)

        // do some business process based on the error type
        if errors.IsCode(err, code.ErrEncodingFailed) {
            fmt.Println("this is a ErrEncodingFailed error")
        }

        if errors.IsCode(err, code.ErrDatabase) {
            fmt.Println("this is a ErrDatabase error")
        }

        // we can also find the cause error
        fmt.Println(errors.Cause(err))
    }
}

func bindUser() error {
    if err := getUser(); err != nil {
        // Step3: Wrap the error with a new error message and a new error code if needed.
        return errors.WrapC(err, code.ErrEncodingFailed, "encoding user 'Lingfei Kong' failed.")
    }

    return nil
}

func getUser() error {
    if err := queryDatabase(); err != nil {
        // Step2: Wrap the error with a new error message.
        return errors.Wrap(err, "get user failed.")
    }

    return nil
}

func queryDatabase() error {
    // Step1. Create error with specified error code.
    return errors.WithCode(code.ErrDatabase, "user 'Lingfei Kong' not found.")
}

In the above code, a new error of type withCode is created using the WithCode function. The WrapC function is used to wrap an existing error into a withCode error. The IsCode function is used to check if a specific error code is present in the error chain.

The withCode error type implements a func (w *withCode) Format(state fmt.State, verb rune) method, which is used to print error messages in different formats. The formats are described in the table below:

For example, %+v will print the following error message:

get user failed. - #1 [/home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/errortrack_errors.go:19 (main.getUser)] (100101) Database error; user 'Lingfei Kong' not found. - #0 [/home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/errortrack_errors.go:26 (main.queryDatabase)] (100101) Database error

Now you may wonder where the error code 100101 and the external error message Database error come from. Let me explain.

Firstly, the withCode struct contains an integer error code, such as 100101.

Secondly, when using the github.com/marmotedu/errors package, you need to call Register or MustRegister to register a Coder in the memory allocated by github.com/marmotedu/errors. The data structure looks like this:

var codes = map[int]Coder{}

Coder is an interface defined as follows:

type Coder interface {
    // HTTPStatus returns the associated HTTP status for the error code.
    HTTPStatus() int
 
    // String returns the user-facing error text.
    String() string
 
    // Reference returns the detailed documentations for the user.
    Reference() string
 
    // Code returns the code of the coder.
    Code() int
}

This way, the Format method of withCode can retrieve the corresponding Coder based on the code field in the withCode struct. It then uses the HTTPStatus, String, Reference, and Code functions provided by Coder to retrieve detailed information about the code in the withCode error and format the output.

It is worth noting that we have implemented two registration functions: Register and MustRegister. The only difference between them is that when the same error code is defined multiple times, MustRegister will panic. This prevents later registered errors from overwriting previously registered errors. In actual development, it is recommended to use MustRegister.

The naming convention of XXX() and MustXXX() is a common Go code design practice. It is often used in Go code, such as the Compile and MustCompile functions provided by the regexp package in the Go standard library. Compared to XXX, using MustXXX implies that a panic may occur if used improperly.

Finally, I have a suggestion: in actual production environments, it is recommended to use JSON-format logs, which can be easily parsed by log systems. You can choose between %#-v and %#+v formats depending on your needs.

Error handling code is often called frequently, so we need to ensure that the error package has high performance to avoid affecting the performance of the interface. Let’s take a look at the performance of the github.com/marmotedu/errors package.

Here, we compare the performance of the github.com/marmotedu/errors package with the errors package from the Go standard library and the github.com/pkg/errors package. Here are the performance benchmarks:

$ go test -test.bench=BenchmarkErrors -benchtime="3s"
goos: linux
goarch: amd64
pkg: github.com/marmotedu/errors
BenchmarkErrors/errors-stack-10-8         	57658672	        61.8 ns/op	      16 B/op	       1 allocs/op
BenchmarkErrors/pkg/errors-stack-10-8     	 2265558	      1547 ns/op	     320 B/op	       3 allocs/op
BenchmarkErrors/marmot/errors-stack-10-8  	 1903532	      1772 ns/op	     360 B/op	       5 allocs/op
BenchmarkErrors/errors-stack-100-8        	 4883659	       734 ns/op	      16 B/op	       1 allocs/op
BenchmarkErrors/pkg/errors-stack-100-8    	 1202797	      2881 ns/op	     320 B/op	       3 allocs/op
BenchmarkErrors/marmot/errors-stack-100-8 	 1000000	      3116 ns/op	     360 B/op	       5 allocs/op
BenchmarkErrors/errors-stack-1000-8       	  505636	      7159 ns/op	      16 B/op	       1 allocs/op
BenchmarkErrors/pkg/errors-stack-1000-8   	  327681	     10646 ns/op	     320 B/op	       3 allocs/op
BenchmarkErrors/marmot/errors-stack-1000-8         	  304160	     11896 ns/op	     360 B/op	       5 allocs/op
PASS

We can see that the performance of the github.com/marmotedu/errors package is comparable to the github.com/pkg/errors package. When comparing performance, the focus is on the ns/op value, which represents the number of nanoseconds per operation. We also need to test performance for different depths of error nesting, as deeper nesting leads to worse performance. For example, at a nesting depth of 10, the ns/op value for the github.com/pkg/errors package is 1547, while for the github.com/marmotedu/errors package, it is 1772. This shows that the performance of both packages is similar.

For more specific performance comparison data, refer to the table below:

We used the BenchmarkErrors test function in the BenchmarkErrors package to test the performance of the error package. If you’re interested, you can open the link to see the details.

How to Record Errors? #

Above, we have discussed how to design an excellent error package. Now, let’s talk about how to use our designed error package to record errors.

Based on my development experience, I recommend two ways to record errors, which can help you quickly locate the issue.

Way 1: Track errors using the error stack capability provided by the github.com/marmotedu/errors package.

You can take a look at the code example below. The following code is saved in errortrack_errors.go.

package main

import (
	"fmt"

	"github.com/marmotedu/errors"

	code "github.com/marmotedu/sample-code"
)

func main() {
	if err := getUser(); err != nil {
		fmt.Printf("%+v\n", err)
	}
}

func getUser() error {
	if err := queryDatabase(); err != nil {
		return errors.Wrap(err, "get user failed.")
	}

	return nil
}

func queryDatabase() error {
	return errors.WithCode(code.ErrDatabase, "user 'Lingfei Kong' not found.")
}

To execute the above code:

$ go run errortrack_errors.go
get user failed. - #1 [/home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/errortrack_errors.go:19 (main.getUser)] (100101) Database error; user 'Lingfei Kong' not found. - #0 [/home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/errors/errortrack_errors.go:26 (main.queryDatabase)] (100101) Database error

As you can see, the printed log contains detailed error stack information, including the function, file name, line number, and error message where the error occurred. With this error stack information, we can easily locate the problem.

When you use this method, I recommend creating a withCode type error using errors.WithCode() at the beginning of the error. In the upper layers, when handling the error returned by the lower layer, you can use the Wrap function to wrap the error with new error messages as needed. If the error to be wrapped is not created using the github.com/marmotedu/errors package, it is recommended to create a new error using errors.WithCode().

Way 2: Call the log package’s logging function at the original location where the error occurred to print the error message, and directly return from other locations (of course, you can optionally add additional error information for fault diagnosis). The following code example (saved in errortrack_log.go) demonstrates this.

package main

import (
	"fmt"

	"github.com/marmotedu/errors"
	"github.com/marmotedu/log"

	code "github.com/marmotedu/sample-code"
)

func main() {
	if err := getUser(); err != nil {
		fmt.Printf("%v\n", err)
	}
}

func getUser() error {
	if err := queryDatabase(); err != nil {
		return err
	}

	return nil
}

func queryDatabase() error {
	opts := &log.Options{
		Level:            "info",
		Format:           "console",
		EnableColor:      true,
		EnableCaller:     true,
		OutputPaths:      []string{"test.log", "stdout"},
		ErrorOutputPaths: []string{},
	}

	log.Init(opts)
	defer log.Flush()

	err := errors.WithCode(code.ErrDatabase, "user 'Lingfei Kong' not found.")
	if err != nil {
		log.Errorf("%v", err)
	}
	return err
}

To execute the above code:

$ go run errortrack_log.go
2021-07-03 14:37:31.597	ERROR	errors/errortrack_log.go:41	Database error
Database error

When an error occurs, call the log package to print the error. With the caller feature of the log package, we can locate the position of the log statement, which is where the error occurred. When printing logs using this method, I have two suggestions:

• Only print logs at the original location where the error occurred, and directly return the error from other places without the need for wrapping.

• When calling a function from a third-party package, print the error information when the function in the third-party package encounters an error. For example:

if err := os.Chdir("/root"); err != nil {
    log.Errorf("change dir failed: %v", err)
}

Implementation of an Error Code #

Next, let’s take a look at a specific implementation of an error code based on the error code specification introduced in the previous lesson github.com/marmotedu/sample-code.

sample-code implements two types of error codes, namely general error codes (sample-code/base.go) and error codes related to business modules (sample-code/apiserver.go).

First, let’s look at the definition of the general error codes:

// General: basic errors
// Code must start with 1xxxxx
const (
    // ErrSuccess - 200: OK.
    ErrSuccess int = iota + 100001

    // ErrUnknown - 500: Internal server error.
    ErrUnknown

    // ErrBind - 400: Error occurred while binding the request body to the struct.
    ErrBind

    // ErrValidation - 400: Validation failed.
    ErrValidation

    // ErrTokenInvalid - 401: Token invalid.
    ErrTokenInvalid
)

In the code, we usually use integer constants (ErrSuccess) instead of integer error codes (100001) because when using ErrSuccess, it is clear what type of error it represents and it is convenient for developers to use.

An error code is used to refer to an error type, and this error type needs to include some useful information, such as the corresponding HTTP Status Code, the Message displayed externally, and the help documentation that matches the error. So, we also need to implement a Coder to carry this information. Here, we define a structure ErrCode that implements the github.com/marmotedu/errors.Coder interface:

// ErrCode implements `github.com/marmotedu/errors`.Coder interface.
type ErrCode struct {
    // C refers to the code of the ErrCode.
    C int

    // HTTP status that should be used for the associated error code.
    HTTP int

    // External (user) facing error text.
    Ext string

    // Ref specify the reference document.
    Ref string
}

As we can see, the ErrCode structure contains the following information:

• An int type business code.
• The corresponding HTTP Status Code.
• The message exposed to external users.
• The reference documentation for the error.

Here is a specific example of a Coder:

coder := &ErrCode{
    C:    100001,
    HTTP: 200,
    Ext:  "OK",
    Ref:  "https://github.com/marmotedu/sample-code/blob/master/README.md",
}

Next, we can call the Register or MustRegister functions provided by the github.com/marmotedu/errors package to register the Coder in the memory maintained by the github.com/marmotedu/errors package.

A project may have many error codes. It would be cumbersome to manually call the MustRegister function for each error code. Here, we use a code generation method to generate the register function calls:

...

//go:generate codegen -type=int
//go:generate codegen -type=int -doc -output ./error_code_generated.md

The //go:generate codegen -type=int will call the codegen tool to generate the sample_code_generated.go source code file:

func init() {
    register(ErrSuccess, 200, "OK")
    register(ErrUnknown, 500, "Internal server error")
    register(ErrBind, 400, "Error occurred while binding the request body to the struct")
    register(ErrValidation, 400, "Validation failed")
    // other register function call
}

These register function invocations are placed in the init function and are initialized when the program is loaded.

It is worth noting that when registering, we check the HTTP Status Code and only allow the definition of 200, 400, 401, 403, 404, and 500 as HTTP error codes. This ensures that the error codes comply with the requirements for using HTTP Status Codes.

The //go:generate codegen -type=int -doc -output ./error_code_generated.md will generate the error code description document error_code_generated.md. When providing API documentation, it is also necessary to provide an error code description document so that the client can determine whether the request is successful and the specific type of error that occurred, and then make some targeted logical processing based on that information.

The codegen tool generates the sample_code_generated.go and error_code_generated.md files based on the error code comments:

// ErrSuccess - 200: OK.
ErrSuccess int = iota + 100001

To install the codegen tool, run the following command in the IAM project root directory:

$ make tools.install.codegen

After installing the codegen tool, you can execute the go generate command in the root directory of the github.com/marmotedu/sample-code package to generate the sample_code_generated.go and error_code_generated.md files. Here’s a tip you should note: it is recommended to name the generated files using the format xxxx_generated.xx. By using generated, we can understand that these files are generated automatically, which helps us understand and use them.

In actual development, we can extract the error codes into a separate package and place it in the internal/pkg/code/ directory for easy application-wide access. For example, IAM error codes are stored in the following files under the IAM project root directory’s internal/pkg/code/ directory:

$ ls base.go apiserver.go authzserver.go 
apiserver.go  authzserver.go  base.go

An application usually includes multiple services. For example, the IAM application includes the iam-apiserver, iam-authz-server, and iam-pump services. Some error codes are shared among these services, so to facilitate maintenance, we can place these shared error codes in the base.go source code file. Other error codes specific to each service can be placed in separate files: iam-apiserver error codes can be put in the apiserver.go file, and iam-authz-server error codes can be put in the authzserver.go file. Similarly for other services.

In addition, for different modules within the same service, we can organize the error codes as follows: put the error codes for the same module in the same const block, and put the error codes for different modules in different const blocks. The beginning comment of each const block can be the error code definition for that module. For example:

// iam-apiserver: user errors.
const (
    // ErrUserNotFound - 404: User not found.
    ErrUserNotFound int = iota + 110001

    // ErrUserAlreadyExist - 400: User already exist.
    ErrUserAlreadyExist
)

// iam-apiserver: secret errors.
const (
    // ErrEncrypt - 400: Secret reach the max count.
    ErrReachMaxCount int = iota + 110101

    //  ErrSecretNotFound - 404: Secret not found.
    ErrSecretNotFound
)

Finally, we need to document the error code definitions in the project file for developers to consult, comply with, and use. For example, the error code definitions for the IAM project are documented in the code_specification.md file. This document includes error code explanations, error description specifications, and error recording specifications, among other things.

Example of Actual Use of Error Codes #

Earlier, I explained the implementation of error packages and error codes. Now you must be wondering how we actually use them in development. Here, I will provide an example of how to use error codes in the gin web framework:

// Response defines project response format which in marmotedu organization.
type Response struct {
    Code      errors.Code `json:"code,omitempty"`
    Message   string      `json:"message,omitempty"`
    Reference string      `json:"reference,omitempty"`
    Data      interface{} `json:"data,omitempty"`
}

// WriteResponse is used to write an error and JSON data into the response.
func WriteResponse(c *gin.Context, err error, data interface{}) {
    if err != nil {
        coder := errors.ParseCoder(err)

        c.JSON(coder.HTTPStatus(), Response{
            Code:      coder.Code(),
            Message:   coder.String(),
            Reference: coder.Reference(),
            Data:      data,
        })
    }

    c.JSON(http.StatusOK, Response{Data: data})
}

func GetUser(c *gin.Context) {
    log.Info("get user function called.", "X-Request-Id", requestid.Get(c))
    // Get the user by the `username` from the database.
    user, err := store.Client().Users().Get(c.Param("username"), metav1.GetOptions{})
    if err != nil {
        core.WriteResponse(c, errors.WithCode(code.ErrUserNotFound, err.Error()), nil)
        return
    }

    core.WriteResponse(c, nil, user)
}

In the above code, the errors are handled uniformly through the WriteResponse function. Inside the WriteResponse function, if err != nil, the Coder is parsed from the error, and the methods provided by the Coder are called to retrieve the HTTP status code, the business code as an integer, the information exposed to the user, and the reference to the error documentation. This information is then returned in JSON format. If err == nil, the function returns 200 and the data.

Summary #

Recording errors is something that an application must do. In actual development, we usually encapsulate our own error packages. An excellent error package should be able to support error stack, different printing formats, Wrap/Unwrap/Is/As functions, and support formatted creation of errors.

Based on these design principles of error packages, I designed the error package github.com/marmotedu/errors for the IAM project, which complies with the error code specification we discussed in the previous lecture.

In addition, this lecture also provides a specific implementation of error codes called sample-code. sample-code supports business codes, HTTP status codes, error reference documents, and can display different error messages internally and externally.

Finally, because the comments for error codes have a fixed format, we can use a codegen tool to parse the comments for error codes and generate register function calls and error code documentation. This approach also reflects the low code thinking that I have been emphasizing, which can improve development efficiency and reduce human errors.

Exercise after class #

1. In this course, we have defined error codes for the base and iam-apiserver services. Please try to define error codes for the iam-authz-server service and generate an error code document.
2. Think about whether the error package and error code design in this course can meet the requirements of your current project. If you feel that it cannot meet your needs, you can share your thoughts in the comments section.

Feel free to discuss and communicate with me in the comments section. See you in the next lecture.