51 Ci Practices Based on Git Hub Actions

51 CI Practices Based on GitHub Actions #

Hello, I’m Kong Lingfei. This is the last lesson of this column. Congratulations on making it to the end!

In Go project development, we often need to perform operations such as static code analysis, testing, compiling, and building. If we manually execute each step, not only will it be inefficient, but it will also be prone to errors. Therefore, we usually use CI systems to automate these operations.

There are many excellent CI systems to choose from in the industry, such as CircleCI, TravisCI, Jenkins, CODING, GitHub Actions, and so on. These systems are similar in design, but to reduce your learning cost, I have chosen GitHub Actions, which is relatively easy to practice, to show you how to automate your work through CI.

In this lesson, I will first introduce GitHub Actions and how to use it, then show you an example of CI, and finally demonstrate how IAM builds CI tasks.

Basic Usage of GitHub Actions #

GitHub Actions is a continuous integration service provided by GitHub for projects hosted on github.com. It was launched in October 2018.

GitHub Actions has the following features:

  • Provides the ability to configure atomic actions and combine actions in workflows.
  • Uses global YAML configuration to be compatible with mainstream CI/CD tool configuration.
  • Supports event triggering for actions/workflows, including event restrictions, webhook events, scheduled events, and external events.
  • Provides managed container services for running workflows, including Docker and VM, with support for Linux, macOS, and Windows.
  • Supports mainstream programming languages, including Node.js, Python, Java, Ruby, PHP, Go, Rust, and .NET.
  • Offers real-time log streams for easy debugging.
  • Provides a wide range of built-in Actions and third-party Actions that are ready to use.

Basic Concepts of GitHub Actions #

When building continuous integration tasks, various operations are performed in the task center, such as cloning code, compiling code, running unit tests, building and publishing images, etc. GitHub refers to these operations as Actions.

Actions can be shared across many projects, and GitHub allows developers to upload these shared Actions to the official GitHub Actions Marketplace, where developers can find Actions submitted by others. Additionally, there is an awesome actions repository that contains many Actions for developers to use. If you need a certain Action, there is no need to write complex scripts yourself. You can simply reference the ready-made Action created by others. The entire continuous integration process becomes a combination of Actions.

An Action is actually an independent script that can be stored in a GitHub code repository and referenced using the <userName>/<repoName> syntax. For example, actions/checkout@v2 represents the repository https://github.com/actions/checkout with the tag v2. actions/checkout@v2 also represents an Action that installs the Go compilation environment. GitHub’s official Actions are located in github.com/actions.

GitHub Actions has its own terminology, which I will introduce below.

  • workflow: One .yml file corresponds to one workflow, which represents a single continuous integration process. A GitHub repository can contain multiple workflows, and any .yml file in the .github/workflow directory will be executed by GitHub.
  • job: A workflow consists of one or more jobs, with each job representing a continuous integration task.
  • step: Each job consists of multiple steps, which are completed step by step.
  • action: Each step can execute one or more commands (actions) sequentially.
  • on: The trigger condition for a workflow, which determines when the workflow should be executed.

Introduction to Workflow Files #

GitHub Actions configuration files are stored in the .github/workflows directory of the code repository, with a .yml file extension. Multiple files can be created, and the file names can be arbitrary, such as iam.yml. Whenever GitHub detects .yml files in the .github/workflows directory, it will automatically run the files. If there are any problems during the execution, you will be notified by email.

Workflow files have many configuration fields. If you want to learn more, you can refer to the official documentation. Here, I will explain some basic configuration fields.

  1. name

The name field represents the name of the workflow. If this field is omitted, the default name is the file name of the current workflow.

name: GitHub Actions Demo
  1. on

The on field specifies the trigger conditions for the workflow, which are usually events.

on: push

The above configuration means that the workflow is triggered by the push event. The on field can also be an array of events, for example:

on: [push, pull_request]

The above configuration means that the workflow can be triggered by either a push event or a pull_request event.

If you want to see the complete list of events, you can refer to the official documentation. In addition to repository events, GitHub Actions also supports external events or scheduled runs.

  1. on.<push|pull_request>.<tags|branches>

When specifying trigger events, you can limit the branches or tags.

on:
  push:
    branches:
      - master

The above configuration specifies that the workflow is only triggered when a push event occurs on the master branch.

  1. jobs.<job_id>.name

The main body of the workflow file is the jobs field, which represents one or more tasks to be executed.

Within the jobs field, you need to specify the job_id for each task, and the name field can be customized as an explanation for the task.

jobs:
  my_first_job:
    name: My first job
  my_second_job:
    name: My second job

The above code contains two tasks in the jobs field, with the job_id being my_first_job and my_second_job respectively.

  1. jobs.<job_id>.needs

The needs field specifies the dependencies and running order of the current task.

jobs:
  job1:
  job2:
    needs: job1
  job3:
    needs: [job1, job2]

In the code above, job1 must be completed before job2, and job3 waits for both job1 and job2 to be completed before running. Therefore, the execution order of this workflow is: job1, job2, job3.

  1. jobs.<job_id>.runs-on

The runs-on field specifies the virtual machine environment needed to run, and it is a required field. The available virtual machines are as follows:

  • ubuntu-latest, ubuntu-18.04, or ubuntu-16.04.
  • windows-latest, windows-2019, or windows-2016.
  • macOS-latest or macOS-10.14.

The following configuration specifies the virtual machine environment as ubuntu-18.04.

runs-on: ubuntu-18.04
  1. jobs.<job_id>.steps

The steps field specifies the run steps for each job, which can include one or more steps. Each step can specify the following three fields:

  • jobs.<job_id>.steps.name: Step name.
  • jobs.<job_id>.steps.run: The command or action to run for this step.
  • jobs.<job_id>.steps.env: The environment variables required for this step.

Here is an example of a complete workflow file:

name: Greeting from Mona
on: push

jobs:
  my-job:
    name: My Job
    runs-on: ubuntu-latest
    steps:
    - name: Print a greeting
      env:
        MY_VAR: Hello! My name is
        FIRST_NAME: Lingfei
        LAST_NAME: Kong
      run: |
        echo $MY_VAR $FIRST_NAME $LAST_NAME.

In the code above, the steps field only includes one step. This step injects three environment variables and then executes a Bash command.

  1. uses

uses can reference actions created by others, which are the actions in the actions marketplace mentioned earlier. The reference format is userName/repoName@version, for example uses: actions/setup-go@v1.

  1. with

with specifies the input parameters for actions. Each input parameter is a key/value pair. The input parameters are set as environment variables with the prefix INPUT_ and converted to uppercase.

Here’s an example: We define three input parameters (first_name, middle_name, and last_name) for the hello_world action, and these input variables will be used as INPUT_FIRST_NAME, INPUT_MIDDLE_NAME, and INPUT_LAST_NAME environment variables by the hello-world action.

jobs:
  my_first_job:
    steps:
      - name: My first step
        uses: actions/hello_world@master
        with:
          first_name: Lingfei
          middle_name: Go
          last_name: Kong
  1. run

run specifies the command(s) to be executed. There can be multiple commands, for example:

- name: Build
  run: |
    go mod tidy
    go build -v -o helloci .
  1. id

id is the unique identifier for a step.

Advanced Usage of GitHub Actions #

Above, I introduced some basic knowledge of GitHub Actions. Here, I will introduce advanced usage of GitHub Actions.

Adding a Badge to the Workflow #

In the Actions panel, click on “Create status badge” to copy the Markdown content of the badge to README.md.

Afterward, we can directly see the current build result in README.md:

Image

Using Matrix Builds #

If we want to test builds on multiple systems or multiple language versions, we need to set up a matrix build. For example, if we want to run tests on multiple operating systems and multiple Go versions, we can use the following workflow configuration:

name: Go Test

on: [push, pull_request]

jobs:

  helloci-build:
    name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        go_version: [1.15, 1.16]
        os: [ubuntu-latest, macOS-latest]

    steps:

      - name: Set up Go ${{ matrix.go_version }}
        uses: actions/setup-go@v2
        with:
          go-version: ${{ matrix.go_version }}
        id: go

In the above workflow configuration, the strategy.matrix section configures the matrix of environments in which the workflow runs (formatted as go_version.os): ubuntu-latest.1.15, ubuntu-latest.1.16, macOS-latest.1.15, and macOS-latest.1.16. In other words, the workflow will be executed on four different servers with different configurations.

Using Secrets #

During the build process, we might need to use sensitive data like ssh or token. However, we don’t want to expose this data directly in the repository. In this case, we can use secrets.

To create a secrets in the corresponding repository, go to Settings -> Secrets, as shown in the following image:

Image

The usage in the configuration file is as follows:

name: Go Test
on: [push, pull_request]
jobs:
  helloci-build:
    name: Test with go
    runs-on: [ubuntu-latest]
    environment:
      name: helloci
    steps:
      - name: use secrets
        env:
          super_secret: ${{ secrets.YourSecrets }}

The secret name is case-insensitive, so if the name of the created secret is “name”, you can use either secrets.name or secrets.Name for usage. Furthermore, even if you directly use echo to print the secret, the console will only print * to protect the secret. Note that your secret belongs to a specific environment variable, so you need to specify the name of the environment: environment.name. The secrets.YourSecrets in the above workflow configuration belongs to the helloci environment.

Using Artifacts to Store Build Artifacts #

During the build process, we may need to output some build artifacts, such as log files and test results. These artifacts can be stored using GitHub Actions Artifact. You can use action/upload-artifact and download-artifact for related operations with build parameters.

Here, I will use the example of saving the Jest test report as an artifact. The coverage is the test artifact produced after running Jest tests:

steps:
      - run: npm ci
      - run: npm test

      - name: Collect Test Coverage File
        uses: actions/[email protected]
        with:
          name: coverage-output
          path: coverage

After successful execution, we can see the generated artifact in the corresponding action panel:

Image

Practical GitHub Actions #

Earlier, I introduced how to use GitHub Actions. Now let’s get hands-on and go through the six specific steps to use GitHub Actions.

Step 1: Create a test repository.

Log in to the GitHub official website and click on New repository as shown in the following image:

Image

Here, we create a test project called helloci.

Step 2: Clone the new repository and add some files:

$ git clone https://github.com/marmotedu/helloci

You can clone marmotedu/helloci and copy the files inside to the project repository you created.

Step 3: Create the GitHub Actions workflow configuration directory:

$ mkdir -p .github/workflows

Step 4: Create the GitHub Actions workflow configuration.

Create a helloci.yml file in the .github/workflows directory with the following content:

name: Go Test

on: [push, pull_request]

jobs:
  helloci-build:
    name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    environment:
      name: helloci

    strategy:
      matrix:
        go_version: [1.16]
        os: [ubuntu-latest]

    steps:
      - name: Set up Go ${{ matrix.go_version }}
        uses: actions/setup-go@v2
        with:
          go-version: ${{ matrix.go_version }}
        id: go

      - name: Check out code into the Go module directory
        uses: actions/checkout@v2

      - name: Tidy
        run: |
          go mod tidy          

      - name: Build
        run: |
          go build -v -o helloci .          

      - name: Collect main.go file
        uses: actions/[email protected]
        with:
          name: main-output
          path: main.go

      - name: Publish to Registry
        uses: elgohr/Publish-Docker-GitHub-Action@master
        with:
          name: ccr.ccs.tencentyun.com/marmotedu/helloci:beta  # docker image name
          username: ${{ secrets.DOCKER_USERNAME}} # username
          password: ${{ secrets.DOCKER_PASSWORD }} # password
          registry: ccr.ccs.tencentyun.com # Tencent Cloud Registry
          dockerfile: Dockerfile # Dockerfile location
          tag_names: true # if release tag should be used as the docker image's tag

The above workflow file defines that when the GitHub repository has push and pull_request events, the GitHub Actions workflow will be triggered. The workflow includes a job called helloci-build, which contains multiple steps. Each step contains some actions.

The above workflow configuration will execute the following six steps in order:

  1. Set up a Go compilation environment.
  2. Download the source code from marmotedu/helloci.
  3. Add or remove missing dependencies.
  4. Compile the Go source code.
  5. Upload the build artifacts.
  6. Build the image and push it to ccr.ccs.tencentyun.com/marmotedu/helloci:beta.

Step 5: Before pushing the code, we need to create the DOCKER_USERNAME and DOCKER_PASSWORD secrets.

DOCKER_USERNAME saves the username for the Tencent Cloud image service (CCR), and DOCKER_PASSWORD saves the password for CCR. We save these two secrets in the helloci Environments as shown in the following image:

Image

Step 6: Push the project to GitHub to trigger the workflow:

$ git add .
$ git push origin master

Open the Actions tab of our repository, and you will find that the GitHub Actions workflow is being executed:

Image

Once the workflow is finished, click on Go Test to enter the build details page and you will see the build history:

Image

Then, select one of the build records and check its run details (refer to chore: update step name Go Test #10 for more information):

Image

You can see that the Go Test workflow executes six jobs, each job includes the following custom steps:

  1. Set up Go 1.16.
  2. Check out code into the Go module directory.
  3. Tidy.
  4. Build.
  5. Collect main.go file.
  6. Publish to Registry.

Other steps are added by GitHub Actions themselves: Setup Job, Post Check out code into the Go module directory, Complete job. By clicking on each step, you can see their detailed outputs.

IAM GitHub Actions Practice #

Next, let’s take a look at the GitHub Actions practice for the IAM project.

Assuming the IAM project root directory is ${IAM_ROOT}, its workflow configuration file is:

$ cat ${IAM_ROOT}/.github/workflows/iamci.yaml
name: IamCI

on:
  push:
    branchs:
    - '*'
  pull_request:
    types: [opened, reopened]

jobs:

  iamci:
    name: Test with go ${{ matrix.go_version }} on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    environment:
      name: iamci

    strategy:
      matrix:
        go_version: [1.16]
        os: [ubuntu-latest]

    steps:

      - name: Set up Go ${{ matrix.go_version }}
        uses: actions/setup-go@v2
        with:
          go-version: ${{ matrix.go_version }}
        id: go

      - name: Check out code into the Go module directory
        uses: actions/checkout@v2

      - name: Run go modules Tidy
        run: |
          make tidy

      - name: Generate all necessary files, such as error code files
        run: |
          make gen

      - name: Check syntax and styling of go sources
        run: |
          make lint

      - name: Run unit test and get test coverage
        run: |
          make cover

      - name: Build source code for host platform
        run: |
          make build

      - name: Collect Test Coverage File
        uses: actions/[[email protected]](/cdn-cgi/l/email-protection)
        with:
          name: main-output
          path: _output/coverage.out

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build docker images for host arch and push images to registry
        run: |
          make push

The above workflow performs the following steps sequentially:

  1. Set up the Go compilation environment.
  2. Download the IAM project source code.
  3. Add/remove unnecessary Go packages.
  4. Generate all code files.
  5. Perform static code checking on the IAM source code.
  6. Run unit tests and calculate the test coverage.
  7. Compile the code.
  8. Collect build artifacts _output/coverage.out.
  9. Configure the Docker build environment.
  10. Log in to DockerHub.
  11. Build Docker images and push them to DockerHub.

The running history of the IamCI workflow is shown in the following figure:

Image

One of the runs of the IamCI workflow is shown in the following figure:

Image

Summary #

In the development of Go projects, we need to automate tasks that require frequent operations through CI tasks. This not only improves development efficiency but also reduces errors caused by manual operations. In this lecture, I chose the easiest-to-implement GitHub Actions to demonstrate how to build CI tasks.

GitHub Actions supports triggering CI workflows through push events. A CI workflow is actually a combination of multiple tasks that can be executed in parallel. Each task consists of multiple steps, and each step consists of multiple actions. An action is essentially a command or script used to complete the specified task, such as compilation.

Because there is a lot of content in GitHub Actions, this lecture only introduces some core knowledge. For a more detailed GitHub Actions tutorial, you can refer to the official documentation.

After-class Practice #

  1. Use CODING to implement the CI task for IAM, and think about whether there are any essential differences between GitHub Actions and CODING in terms of CI task construction?
  2. In this lecture, we used GitHub Actions to implement CI. Please combine the knowledge you have learned earlier to implement the CD functionality for IAM. Pull Requests are welcome.

This is the last exercise of our course. Please feel free to share your thoughts and ideas in the comments section. You are also welcome to share the course with your colleagues and friends. Let’s communicate and progress together.