51 CI Practices Based on GitHub Actions #
Hello, I’m Kong Lingfei. This is the last lesson of this column. Congratulations on making it to the end!
In Go project development, we often need to perform operations such as static code analysis, testing, compiling, and building. If we manually execute each step, not only will it be inefficient, but it will also be prone to errors. Therefore, we usually use CI systems to automate these operations.
There are many excellent CI systems to choose from in the industry, such as CircleCI, TravisCI, Jenkins, CODING, GitHub Actions, and so on. These systems are similar in design, but to reduce your learning cost, I have chosen GitHub Actions, which is relatively easy to practice, to show you how to automate your work through CI.
In this lesson, I will first introduce GitHub Actions and how to use it, then show you an example of CI, and finally demonstrate how IAM builds CI tasks.
Basic Usage of GitHub Actions #
GitHub Actions is a continuous integration service provided by GitHub for projects hosted on github.com. It was launched in October 2018.
GitHub Actions has the following features:
- Provides the ability to configure atomic actions and combine actions in workflows.
- Uses global YAML configuration to be compatible with mainstream CI/CD tool configuration.
- Supports event triggering for actions/workflows, including event restrictions, webhook events, scheduled events, and external events.
- Provides managed container services for running workflows, including Docker and VM, with support for Linux, macOS, and Windows.
- Supports mainstream programming languages, including Node.js, Python, Java, Ruby, PHP, Go, Rust, and .NET.
- Offers real-time log streams for easy debugging.
- Provides a wide range of built-in Actions and third-party Actions that are ready to use.
Basic Concepts of GitHub Actions #
When building continuous integration tasks, various operations are performed in the task center, such as cloning code, compiling code, running unit tests, building and publishing images, etc. GitHub refers to these operations as Actions.
Actions can be shared across many projects, and GitHub allows developers to upload these shared Actions to the official GitHub Actions Marketplace, where developers can find Actions submitted by others. Additionally, there is an awesome actions repository that contains many Actions for developers to use. If you need a certain Action, there is no need to write complex scripts yourself. You can simply reference the ready-made Action created by others. The entire continuous integration process becomes a combination of Actions.
An Action is actually an independent script that can be stored in a GitHub code repository and referenced using the <userName>/<repoName>
syntax. For example, actions/checkout@v2
represents the repository https://github.com/actions/checkout
with the tag v2
. actions/checkout@v2
also represents an Action that installs the Go compilation environment. GitHub’s official Actions are located in github.com/actions.
GitHub Actions has its own terminology, which I will introduce below.
- workflow: One
.yml
file corresponds to one workflow, which represents a single continuous integration process. A GitHub repository can contain multiple workflows, and any.yml
file in the.github/workflow
directory will be executed by GitHub. - job: A workflow consists of one or more jobs, with each job representing a continuous integration task.
- step: Each job consists of multiple steps, which are completed step by step.
- action: Each step can execute one or more commands (actions) sequentially.
- on: The trigger condition for a workflow, which determines when the workflow should be executed.
Introduction to Workflow Files #
GitHub Actions configuration files are stored in the .github/workflows
directory of the code repository, with a .yml
file extension. Multiple files can be created, and the file names can be arbitrary, such as iam.yml
. Whenever GitHub detects .yml
files in the .github/workflows
directory, it will automatically run the files. If there are any problems during the execution, you will be notified by email.
Workflow files have many configuration fields. If you want to learn more, you can refer to the official documentation. Here, I will explain some basic configuration fields.
name
The name
field represents the name of the workflow. If this field is omitted, the default name is the file name of the current workflow.
name: GitHub Actions Demo
on
The on
field specifies the trigger conditions for the workflow, which are usually events.
on: push
The above configuration means that the workflow is triggered by the push
event. The on
field can also be an array of events, for example:
on: [push, pull_request]
The above configuration means that the workflow can be triggered by either a push
event or a pull_request
event.
If you want to see the complete list of events, you can refer to the official documentation. In addition to repository events, GitHub Actions also supports external events or scheduled runs.
on.<push|pull_request>.<tags|branches>
When specifying trigger events, you can limit the branches or tags.
on:
push:
branches:
- master
The above configuration specifies that the workflow is only triggered when a push
event occurs on the master
branch.
jobs.<job_id>.name
The main body of the workflow file is the jobs
field, which represents one or more tasks to be executed.
Within the jobs
field, you need to specify the job_id
for each task, and the name field can be customized as an explanation for the task.
jobs:
my_first_job:
name: My first job
my_second_job:
name: My second job
The above code contains two tasks in the jobs
field, with the job_id
being my_first_job
and my_second_job
respectively.
jobs.<job_id>.needs
The needs
field specifies the dependencies and running order of the current task.
jobs:
job1:
job2:
needs: job1
job3:
needs: [job1, job2]
In the code above, job1
must be completed before job2
, and job3
waits for both job1
and job2
to be completed before running. Therefore, the execution order of this workflow is: job1
, job2
, job3
.
jobs.<job_id>.runs-on
The runs-on
field specifies the virtual machine environment needed to run, and it is a required field. The available virtual machines are as follows:
- ubuntu-latest, ubuntu-18.04, or ubuntu-16.04.
- windows-latest, windows-2019, or windows-2016.
- macOS-latest or macOS-10.14.
The following configuration specifies the virtual machine environment as ubuntu-18.04
.
runs-on: ubuntu-18.04
jobs.<job_id>.steps
The steps
field specifies the run steps for each job, which can include one or more steps. Each step can specify the following three fields:
jobs.<job_id>.steps.name
: Step name.jobs.<job_id>.steps.run
: The command or action to run for this step.jobs.<job_id>.steps.env
: The environment variables required for this step.
Here is an example of a complete workflow file:
name: Greeting from Mona
on: push
jobs:
my-job:
name: My Job
runs-on: ubuntu-latest
steps:
- name: Print a greeting
env:
MY_VAR: Hello! My name is
FIRST_NAME: Lingfei
LAST_NAME: Kong
run: |
echo $MY_VAR $FIRST_NAME $LAST_NAME.
In the code above, the steps
field only includes one step. This step injects three environment variables and then executes a Bash command.
uses
uses
can reference actions created by others, which are the actions in the actions marketplace mentioned earlier. The reference format is userName/repoName@version
, for example uses: actions/setup-go@v1
.
with
with
specifies the input parameters for actions. Each input parameter is a key/value pair. The input parameters are set as environment variables with the prefix INPUT_
and converted to uppercase.
Here’s an example: We define three input parameters (first_name
, middle_name
, and last_name
) for the hello_world
action, and these input variables will be used as INPUT_FIRST_NAME
, INPUT_MIDDLE_NAME
, and INPUT_LAST_NAME
environment variables by the hello-world
action.
jobs:
my_first_job:
steps:
- name: My first step
uses: actions/hello_world@master
with:
first_name: Lingfei
middle_name: Go
last_name: Kong
run
run
specifies the command(s) to be executed. There can be multiple commands, for example:
- name: Build
run: |
go mod tidy
go build -v -o helloci .
id
id
is the unique identifier for a step.
Advanced Usage of GitHub Actions #
Above, I introduced some basic knowledge of GitHub Actions. Here, I will introduce advanced usage of GitHub Actions.
Adding a Badge to the Workflow #
In the Actions panel, click on “Create status badge” to copy the Markdown content of the badge to README.md.
Afterward, we can directly see the current build result in README.md:
Using Matrix Builds #
If we want to test builds on multiple systems or multiple language versions, we need to set up a matrix build. For example, if we want to run tests on multiple operating systems and multiple Go versions, we can use the following workflow configuration:
name: Go Test
on: [push, pull_request]
jobs:
helloci-build:
name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
go_version: [1.15, 1.16]
os: [ubuntu-latest, macOS-latest]
steps:
- name: Set up Go ${{ matrix.go_version }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go_version }}
id: go
In the above workflow configuration, the strategy.matrix
section configures the matrix of environments in which the workflow runs (formatted as go_version.os
): ubuntu-latest.1.15
, ubuntu-latest.1.16
, macOS-latest.1.15
, and macOS-latest.1.16
. In other words, the workflow will be executed on four different servers with different configurations.
Using Secrets #
During the build process, we might need to use sensitive data like ssh
or token
. However, we don’t want to expose this data directly in the repository. In this case, we can use secrets
.
To create a secrets
in the corresponding repository, go to Settings
-> Secrets
, as shown in the following image:
The usage in the configuration file is as follows:
name: Go Test
on: [push, pull_request]
jobs:
helloci-build:
name: Test with go
runs-on: [ubuntu-latest]
environment:
name: helloci
steps:
- name: use secrets
env:
super_secret: ${{ secrets.YourSecrets }}
The secret name is case-insensitive, so if the name of the created secret is “name”, you can use either secrets.name
or secrets.Name
for usage. Furthermore, even if you directly use echo
to print the secret, the console will only print *
to protect the secret. Note that your secret belongs to a specific environment variable, so you need to specify the name of the environment: environment.name
. The secrets.YourSecrets
in the above workflow configuration belongs to the helloci
environment.
Using Artifacts to Store Build Artifacts #
During the build process, we may need to output some build artifacts, such as log files and test results. These artifacts can be stored using GitHub Actions Artifact. You can use action/upload-artifact and download-artifact for related operations with build parameters.
Here, I will use the example of saving the Jest test report as an artifact. The coverage is the test artifact produced after running Jest tests:
steps:
- run: npm ci
- run: npm test
- name: Collect Test Coverage File
uses: actions/[email protected]
with:
name: coverage-output
path: coverage
After successful execution, we can see the generated artifact in the corresponding action panel:
Practical GitHub Actions #
Earlier, I introduced how to use GitHub Actions. Now let’s get hands-on and go through the six specific steps to use GitHub Actions.
Step 1: Create a test repository.
Log in to the GitHub official website and click on New repository as shown in the following image:
Here, we create a test project called helloci
.
Step 2: Clone the new repository and add some files:
$ git clone https://github.com/marmotedu/helloci
You can clone marmotedu/helloci and copy the files inside to the project repository you created.
Step 3: Create the GitHub Actions workflow configuration directory:
$ mkdir -p .github/workflows
Step 4: Create the GitHub Actions workflow configuration.
Create a helloci.yml
file in the .github/workflows
directory with the following content:
name: Go Test
on: [push, pull_request]
jobs:
helloci-build:
name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
environment:
name: helloci
strategy:
matrix:
go_version: [1.16]
os: [ubuntu-latest]
steps:
- name: Set up Go ${{ matrix.go_version }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go_version }}
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v2
- name: Tidy
run: |
go mod tidy
- name: Build
run: |
go build -v -o helloci .
- name: Collect main.go file
uses: actions/[email protected]
with:
name: main-output
path: main.go
- name: Publish to Registry
uses: elgohr/Publish-Docker-GitHub-Action@master
with:
name: ccr.ccs.tencentyun.com/marmotedu/helloci:beta # docker image name
username: ${{ secrets.DOCKER_USERNAME}} # username
password: ${{ secrets.DOCKER_PASSWORD }} # password
registry: ccr.ccs.tencentyun.com # Tencent Cloud Registry
dockerfile: Dockerfile # Dockerfile location
tag_names: true # if release tag should be used as the docker image's tag
The above workflow file defines that when the GitHub repository has push
and pull_request
events, the GitHub Actions workflow will be triggered. The workflow includes a job called helloci-build
, which contains multiple steps. Each step contains some actions.
The above workflow configuration will execute the following six steps in order:
- Set up a Go compilation environment.
- Download the source code from marmotedu/helloci.
- Add or remove missing dependencies.
- Compile the Go source code.
- Upload the build artifacts.
- Build the image and push it to
ccr.ccs.tencentyun.com/marmotedu/helloci:beta
.
Step 5: Before pushing the code, we need to create the DOCKER_USERNAME
and DOCKER_PASSWORD
secrets.
DOCKER_USERNAME
saves the username for the Tencent Cloud image service (CCR), and DOCKER_PASSWORD
saves the password for CCR. We save these two secrets in the helloci
Environments as shown in the following image:
Step 6: Push the project to GitHub to trigger the workflow:
$ git add .
$ git push origin master
Open the Actions tab of our repository, and you will find that the GitHub Actions workflow is being executed:
Once the workflow is finished, click on Go Test to enter the build details page and you will see the build history:
Then, select one of the build records and check its run details (refer to chore: update step name Go Test #10 for more information):
You can see that the Go Test
workflow executes six jobs, each job includes the following custom steps:
- Set up Go 1.16.
- Check out code into the Go module directory.
- Tidy.
- Build.
- Collect main.go file.
- Publish to Registry.
Other steps are added by GitHub Actions themselves: Setup Job
, Post Check out code into the Go module directory
, Complete job
. By clicking on each step, you can see their detailed outputs.
IAM GitHub Actions Practice #
Next, let’s take a look at the GitHub Actions practice for the IAM project.
Assuming the IAM project root directory is ${IAM_ROOT}
, its workflow configuration file is:
$ cat ${IAM_ROOT}/.github/workflows/iamci.yaml
name: IamCI
on:
push:
branchs:
- '*'
pull_request:
types: [opened, reopened]
jobs:
iamci:
name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
environment:
name: iamci
strategy:
matrix:
go_version: [1.16]
os: [ubuntu-latest]
steps:
- name: Set up Go ${{ matrix.go_version }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go_version }}
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v2
- name: Run go modules Tidy
run: |
make tidy
- name: Generate all necessary files, such as error code files
run: |
make gen
- name: Check syntax and styling of go sources
run: |
make lint
- name: Run unit test and get test coverage
run: |
make cover
- name: Build source code for host platform
run: |
make build
- name: Collect Test Coverage File
uses: actions/[[email protected]](/cdn-cgi/l/email-protection)
with:
name: main-output
path: _output/coverage.out
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Build docker images for host arch and push images to registry
run: |
make push
The above workflow performs the following steps sequentially:
- Set up the Go compilation environment.
- Download the IAM project source code.
- Add/remove unnecessary Go packages.
- Generate all code files.
- Perform static code checking on the IAM source code.
- Run unit tests and calculate the test coverage.
- Compile the code.
- Collect build artifacts
_output/coverage.out
. - Configure the Docker build environment.
- Log in to DockerHub.
- Build Docker images and push them to DockerHub.
The running history of the IamCI workflow is shown in the following figure:
One of the runs of the IamCI workflow is shown in the following figure:
Summary #
In the development of Go projects, we need to automate tasks that require frequent operations through CI tasks. This not only improves development efficiency but also reduces errors caused by manual operations. In this lecture, I chose the easiest-to-implement GitHub Actions to demonstrate how to build CI tasks.
GitHub Actions supports triggering CI workflows through push events. A CI workflow is actually a combination of multiple tasks that can be executed in parallel. Each task consists of multiple steps, and each step consists of multiple actions. An action is essentially a command or script used to complete the specified task, such as compilation.
Because there is a lot of content in GitHub Actions, this lecture only introduces some core knowledge. For a more detailed GitHub Actions tutorial, you can refer to the official documentation.
After-class Practice #
- Use CODING to implement the CI task for IAM, and think about whether there are any essential differences between GitHub Actions and CODING in terms of CI task construction?
- In this lecture, we used GitHub Actions to implement CI. Please combine the knowledge you have learned earlier to implement the CD functionality for IAM. Pull Requests are welcome.
This is the last exercise of our course. Please feel free to share your thoughts and ideas in the comments section. You are also welcome to share the course with your colleagues and friends. Let’s communicate and progress together.