Special Delivery Go Modules Practice

Special Delivery Go Modules Practice #

Hello, I’m Kong Lingfei.

Today, we have an extra special episode as a bonus. In Special Episode | A Comprehensive Guide to Go Modules Dependency Management, I introduced the knowledge about Go Modules, which contains quite a lot of content. You may not yet know how to use Go Modules to manage Go dependencies for your projects.

In this episode, I will guide you step by step through a specific case study to learn about the common usage and operations of Go Modules. The specific topics covered in this episode are:

  1. Preparing a demonstration project.
  2. Configuring Go Modules.
  3. Initializing Go packages as Go modules.
  4. Go package dependency management.

Prepare a demo project #

To demonstrate the usage of Go Modules, we first need a demo project. Assuming we have a project called “hello” with two files, hello.go and hello_test.go, located at /home/lk/workspace/golang/src/github.com/marmotedu/gopractise-demo/modules/hello.

The content of hello.go is as follows:

package hello

func Hello() string {
    return "Hello, world."
}

The content of hello_test.go is as follows:

package hello

import "testing"

func TestHello(t *testing.T) {
    want := "Hello, world."
    if got := Hello(); got != want {
        t.Errorf("Hello() = %q, want %q", got, want)
    }
}

At this point, the directory contains a Go package, but it is not yet a Go module because it does not have a go.mod file. Next, I’ll show you how to turn this package into a Go module and perform management operations for Go dependencies. There are a total of 10 steps for these operations. Let’s go through them one by one.

Configure Go Modules #

  1. Enable Go Modules

Make sure your Go version is >= go1.11, and enable Go Modules by setting the environment variable export GO111MODULE=on. If you find it cumbersome to set it each time, you can append export GO111MODULE=on to the file $HOME/.bashrc, and execute the bash command to load it into the current shell environment.

  1. Set environment variables

For developers in China, it is necessary to set export GOPROXY=https://goproxy.cn,direct, so that some blocked packages can be installed from the mirror source in China. If you have some modules stored in a private repository, you also need to set the GOPRIVATE environment variable.

Since Go Modules will request the Go Checksum Database, accessing the Checksum Database may fail in China. You can disable checksum verification by setting export GOSUMDB=off. For certain modules, if you do not want to use a proxy server or verify the checksum, you can also set GONOPROXY and GONOSUMDB according to your needs.

Initialize Go package as a Go module #

  1. Create a new module

You can initialize a project as Go Modules using the go mod init command. The init command initializes and creates a new go.mod file in the current directory, representing a Go module rooted in the project’s root directory. If there is already a go.mod file in the current directory, the initialization will fail.

When initializing Go Modules, you need to provide the module name to go mod init. You can specify the module name, for example, go mod init github.com/marmotedu/gopractise-demo/modules/hello. You can also omit the module name, and let init deduce it. Let me explain the deducing rules below.

  • If there is an import path comment, the comment will be used as the module name, for example:
package hello // import "github.com/marmotedu/gopractise-demo/modules/hello"

In this case, the module name will be github.com/marmotedu/gopractise-demo/modules/hello.

  • If there is no import path comment, and the project is under the GOPATH, the module name will be the absolute path without the $GOPATH/src part. For example, if GOPATH=/home/lk/workspace/golang and the absolute path of the project is /home/colin/workspace/golang/src/github.com/marmotedu/gopractise-demo/modules/hello, the module name will be github.com/marmotedu/gopractise-demo/modules/hello.

After the initialization is complete, a go.mod file will be generated in the current directory:

$ cat go.mod
module github.com/marmotedu/gopractise-demo/modules/hello

go 1.14

The content of the file indicates that the import path of the current module is github.com/marmotedu/gopractise-demo/modules/hello, and it is using Go version go 1.14.

If you want to create a new package under a subdirectory, the import path of the package will automatically be module_name/sub-directory_name: github.com/marmotedu/gopractise-demo/modules/hello/<sub-package-name>. You don’t need to execute go mod init again in the subdirectory.

For example, if we create a world package under the hello directory, the import path for the world package will be github.com/marmotedu/gopractise-demo/modules/hello/world.

Go Package Dependency Management #

  1. Adding a Dependency

Go Modules are mainly used for managing package dependencies. So here we are going to add a dependency rsc.io/quote to the hello package:

package hello

import "rsc.io/quote"

func Hello() string {
    return quote.Hello()
}

Run go test:

$ go test
go: finding module for package rsc.io/quote
go: downloading rsc.io/quote v1.5.2
go: found rsc.io/quote in rsc.io/quote v1.5.2
go: downloading rsc.io/sampler v1.3.0
PASS
ok  	github.com/google/addlicense/golang/src/github.com/marmotedu/gopractise-demo/modules/hello	0.003s

When the go command parses the source code and encounters an import of a module, it checks go.mod for the module’s version. If a specific version is specified, that version is imported.

If the module is not found in go.mod, the go command will automatically install the module based on the module’s import path and add it to go.mod, along with its latest version. In our example, go test resolves the module rsc.io/quote to version rsc.io/quote v1.5.2, and it also downloads two dependency modules of rsc.io/quote: rsc.io/quote and rsc.io/sampler. Only direct dependencies are recorded in go.mod.

View the go.mod file:

module github.com/marmotedu/gopractise-demo/modules/hello

go 1.14

require rsc.io/quote v1.5.2

Run go test again:

$ go test
PASS
ok  	github.com/marmotedu/gopractise-demo/modules/hello	0.003s

When we run go test again, it will not download and record the required modules again because go.mod is up to date and the required modules are already cached in the local $GOPATH/pkg/mod directory. Additionally, a go.sum file is generated in the current directory:

$ cat go.sum
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/QiW4=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

The go test command can also be used with the -mod flag, such as go test -mod=vendor. The -mod flag has 3 possible values, which I will explain separately.

  • readonly: It does not update go.mod, and any operation that may change go.mod will fail. It is usually used to check whether go.mod needs to be updated, such as in CI or test scenarios.
  • vendor: It imports packages from the vendor directory in the project’s root instead of from the module cache. Make sure that the vendor package is complete and accurate.
  • mod: It imports packages from the module cache, even if the vendor directory is present in the project’s root directory.

If the go test command is executed without the -mod flag and the vendor directory exists in the project’s root directory, and the go version specified in go.mod is greater than or equal to 1.14, then go test is equivalent to go test -mod=vendor. The -mod flag is also applicable to the go build, go install, go run, go test, go list, and go vet commands.

  1. Viewing All Dependencies

We can use the go list -m all command to view all dependencies:

$ go list -m all
github.com/marmotedu/gopractise-demo/modules/hello
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0

As we can see, besides rsc.io/quote v1.5.2, there are also indirect dependencies on other modules.

  1. Updating Dependencies

Based on the output of go list -m all, we can see that the version of the golang.org/x/text module that our module depends on is v0.0.0. We can use the go get command to update it to the latest version and observe if the tests pass:

$ go get golang.org/x/text
go: golang.org/x/text upgrade => v0.3.3
$ go test
PASS
ok  	github.com/marmotedu/gopractise-demo/modules/hello	0.003s

The successful output of go test indicates that the upgrade was successful. Let’s take a look at go list -m all and the go.mod file again:

$ go list -m all
github.com/marmotedu/gopractise-demo/modules/hello
golang.org/x/text v0.3.3
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e
rsc.io/quote v1.5.2
rsc.io/sampler v1.3.0
$ cat go.mod
module github.com/marmotedu/gopractise-demo/modules/hello

go 1.14

require (
    golang.org/x/text v0.3.3 // indirect
    rsc.io/quote v1.5.2
)

As we can see, the golang.org/x/text package has been updated to the latest tag version (v0.3.3) and go.mod has been updated as well. The // indirect comment indicates that golang.org/x/text is an indirect dependency. Now let’s try updating rsc.io/sampler to a different version and test it:

$ go get rsc.io/sampler
go: rsc.io/sampler upgrade => v1.99.99
go: downloading rsc.io/sampler v1.99.99
$ go test
--- FAIL: TestHello (0.00s)
    hello_test.go:8: Hello() = "99 bottles of beer on the wall, 99 bottles of beer, ...", want "Hello, world."
FAIL
exit status 1
FAIL	github.com/marmotedu/gopractise-demo/modules/hello	0.004s

The test fails, indicating that the latest version v1.99.99 is not compatible with our current module. We can list all available versions of rsc.io/sampler and try updating to another version:

$ go list -m -versions rsc.io/sampler
rsc.io/sampler v1.0.0 v1.2.0 v1.2.1 v1.3.0 v1.3.1 v1.99.99

# Let's try selecting a second-to-newest version v1.3.1

$ go get rsc.io/[email protected]
go: downloading rsc.io/sampler v1.3.1
$ go test
PASS
ok  	github.com/marmotedu/gopractise-demo/modules/hello	0.004s

As you can see, we have updated to version v1.3.1 and all tests have passed. `go get` also supports various parameters, as shown in the following table:

| Parameter  | Description                                  |
|------------|----------------------------------------------|
| `@latest`  | Get the latest version of the package         |
| `@<version>` | Get a specific version of the package        |
| `@<branch>`  | Get a specific branch of the package       |
| `@<commit>` | Get a specific commit of the package       |


1. Adding a new major version dependency

Let's try adding a new function `func Proverb` that uses the `quote.Concurrency` function from `rsc.io/quote/v3`.

First, we add the new function in the hello.go file:

```go
package hello

import (
    "rsc.io/quote"
    quoteV3 "rsc.io/quote/v3"
)

func Hello() string {
    return quote.Hello()
}

func Proverb() string {
    return quoteV3.Concurrency()
}

Next, we add a test case for this function in hello_test.go:

func TestProverb(t *testing.T) {
    want := "Concurrency is not parallelism."
    if got := Proverb(); got != want {
        t.Errorf("Proverb() = %q, want %q", got, want)
    }
}

Then, we run the tests:

$ go test
go: finding module for package rsc.io/quote/v3
go: found rsc.io/quote/v3 in rsc.io/quote/v3 v3.1.0
PASS
ok  	github.com/marmotedu/gopractise-demo/modules/hello	0.003s

The tests passed. We can see that the current module depends on two different versions of the same module, rsc.io/quote and rsc.io/quote/v3:

$ go list -m rsc.io/q...
rsc.io/quote v1.5.2
rsc.io/quote/v3 v3.1.0
  1. Upgrading to an incompatible version

In the previous step, we used the Hello() function from rsc.io/quote v1. According to semantic versioning rules, if we want to upgrade the major version, we may face interface incompatibility and need to modify our code. Let’s take a look at the functions in rsc.io/quote/v3:

$ go doc rsc.io/quote/v3
package quote // import "github.com/google/addlicense/golang/pkg/mod/rsc.io/quote/[email protected]"

Package quote collects pithy sayings.

func Concurrency() string
func GlassV3() string
func GoV3() string
func HelloV3() string
func OptV3() string

We can see that the Hello() function has been renamed to HelloV3(), so we need to modify our code to adapt to this change. Since we have unified the modules to the same version, we don’t need to rename the module to avoid conflicts. Therefore, the updated hello.go file looks like this:

package hello

import (
    "rsc.io/quote/v3"
)

func Hello() string {
    return quote.HelloV3()
}

func Proverb() string {
    return quote.Concurrency()
}

We can then run go test:

$ go test
PASS
ok  	github.com/marmotedu/gopractise-demo/modules/hello	0.003s

The tests passed successfully.

  1. Removing unused dependencies

In the previous step, we removed the rsc.io/quote package, but it still exists in go list -m all and go.mod. To clean up the unused dependencies, we can run go mod tidy:

$ go mod tidy
[colin@dev hello]$ cat go.mod
module github.com/marmotedu/gopractise-demo/modules/hello

go 1.14

require (
    golang.org/x/text v0.3.3 // indirect
    rsc.io/quote/v3 v3.1.0
    rsc.io/sampler v1.3.1 // indirect
)
  1. Using vendor

If we want to save all dependencies and avoid downloading them again when running the Go command, we can use go mod vendor. This command will save all dependencies of the current project in the vendor directory in the root of the project. It will also create a vendor/modules.txt file to record the package and module versions:

$ go mod vendor
$ ls
go.mod  go.sum  hello.go  hello_test.go  vendor  world

That’s it for the 10 common operations in Go dependency management.

## Summary

In this lecture, I have detailed how to use Go Modules to manage dependencies. It includes the following operations of Go Modules:

  1. Enable Go Modules；
  2. Set environment variables；
  3. Create a new module；
  4. Add a dependency；
  5. View all dependent modules；
  6. Update dependencies；
  7. Add a new major version dependency；
  8. Upgrade to an incompatible version；
  9. Remove unused dependencies.
  10. Use vendor.
## Homework Exercises

1. Consider how to update all dependencies of a project to their latest versions.

2. Consider how to download Go dependency packages through Go Modules if our build machine cannot access the Internet.

Feel free to discuss and communicate with me in the comment section. See you in the next lesson.