15 Introduction and Detailed Syntax of Jenkins Kubernetes Plugin

15Introduction and Detailed Syntax of Jenkins Kubernetes Plugin #

In the previous section, we introduced different installation methods for configuring Jenkins to connect to a Kubernetes cluster, as well as a simple example. Depending on the purpose of the Kubernetes plugin, the syntax of the plugin will be explained differently. In this section, we will mainly introduce the Kubernetes plugin that is used to dynamically generate Jenkins slave pods in a Kubernetes cluster.

Kubernetes plugin #

When writing pipeline scripts to integrate the Kubernetes plugin and generate dynamic slaves, the code for starting the agent proxy will vary depending on the type of pipeline syntax being used. However, regardless of the differences, there is one core method that remains the same, which is the PodTemplate. As the name suggests, it is a template for the pod.

The PodTemplate is the core method used by the kubernetes plugin to generate dynamic slave nodes. It is used to configure the resource object definition for the agent proxy (Pod). For those familiar with Kubernetes, you should know that the definition of a Kubernetes resource object, the pod, mainly consists of two parts: the configuration definition of the pod and the configuration definition of the container. In Jenkins, these two parts are defined and the agent is started using the PodTemplate() method provided by the Kubernetes plugin.

Below, we will introduce the configuration methods of the PodTemplate based on different syntax types.

Basic Syntax #

Scripting Syntax #

First, let’s look at the basic syntax for using the Kubernetes plugin in a script-based pipeline.

podTemplate(label: 'xxx', cloud: '', <pod_option1>, <pod_option2>, ..., containers: [
    <containerTemplate(name: 'c1', <container_option1>, <container_option2>, ...)>,
    <containerTemplate(name: 'c2', <container_option1>, <container_option2>, ...)>,
    ....
]){
    node('xxx'){
        stage('test'){
            ....
        }
    }
    container('c1') {
        stage('c1'){
            ....
        }
    }
    container('c2') {
        stage('c2'){
            ....
        }
    }
}

Explanation:

The podTemplate consists of two parts:

The first part defines some properties of the Kubernetes Pod resource itself, such as labels, pod name, cloud name, and mounted volumes.

The second part defines the containers within the pod. The containers are listed under the containers property and can include one or more containerTemplates. Each containerTemplate describes the configuration parameters of a container, such as image address, container name, and pull policy.

podTemplate() defines a unique label name, which is used to reference the pod within node(), in order to generate a unique pod name.

containerTemplate() defines the name of the container, which is used as the execution environment for the pipeline. It can be referenced within the pipeline using container('container_name').

Declarative Syntax #

Compared to the script-based syntax, the declarative syntax may appear more confusing. This is because the fundamental configuration of declarative syntax involves directly placing the contents of the resource object “pod” in Kubernetes into the pipeline script using YAML, as shown below (only partial definition is shown).

pipeline {
    agent {
        kubernetes {
            label <label_name>
            yaml '''
kind: Pod
metadata:
  name: <pod_name>
spec:
  containers:
  - name: <container_name>
    image: <image_name>
    imagePullPolicy: IfNotPresent
    command:
    - xxxx
    tty: true
    volumeMounts:
      ....
  restartPolicy: Never
  volumes:
    ......
'''
       }
    }
    stages {
        stage('one') {
          steps {
            container('container_name') {
              <command1>
            }
            container('container_name') {
              <command2>
            }
          }
          steps {
          sh 'hostname'
        }
   }       
}

Explanation

The agent{} section uses the kubernetes keyword to directly include the contents of the Pod definition YAML file into the PodTemplate.

The reference to the container in different stages also uses the syntax container('container_name').

The above code provides the basic syntax for dynamically generating Jenkins slave nodes using the Kubernetes plugin, integrating script-based and declarative pipelines. It should be noted that the Pod Template and Container Template configurations in the PodTemplate method can be configured in the standard Jenkins UI, or in the pipeline script using scripts. This section mainly describes how to define PodTemplate using scripts in the pipeline script. The method of configuring PodTemplate in the Jenkins UI will be covered later, and if PodTemplate is defined using scripts, the configuration process in the Jenkins UI will be easier.

Parameter Description #

Since PodTemplate() is a method, it will definitely involve parameter configuration. Therefore, the following section will first introduce the parameter configuration of the PodTemplate() method. To differentiate between pod configuration and container configuration, I will split the overall configuration of PodTemplate() into a Pod Template configuration and a container Template configuration.

Parameters of the Pod Template Configuration:

  • cloud: Used to specify the cloud name set in the Jenkins system configuration, defaulting to “kubernetes”. This parameter was explained in the previous section and will not be repeated here.

  • name: The name of the pod.

  • namespace: The namespace in which the pod runs.

  • label: The label of the pod. It can be self-defined or use the built-in POD_LABEL variable provided by the plugin.

  • yaml: The YAML representation of the pod definition. You can refer to the official Kubernetes documentation for more details.

  • yamlMergeStrategy: It includes the merge() and override() properties. It controls whether the YAML definition overrides or merges with the YAML definition of the declared (or inherited) pod template. The default value is override().

  • containers: The definition of the pod’s container template.

  • serviceAccount: The service account used by the pod.

  • nodeSelector: The node on which the generated pod is bound, expressed as key: value.

  • nodeUsageMode: It includes two values: NORMAL and EXCLUSIVE. It controls how Jenkins selects this agent: whether to only use this node when the node label is specified, or to use this node as much as possible.

  • volumes: The defined volumes used by the pod and all containers.

  • envVars: The environment variables applied to all containers.

  • envVar: The environment variable defined within the pipeline.

  • secretEnvVar: Secret variables whose values are obtained from Kubernetes secrets.

  • imagePullSecrets: A secret that stores authentication information for a private repository, used to authenticate with the private repository when pulling images.

  • annotations: Annotations for the pod.

  • InheritFrom: A list of inherited pod templates.

  • slaveConnectTimeout: The timeout (in seconds) for the agent pod to connect to Jenkins.

  • podRetention: Controls the behavior of retaining the pod, which determines whether the pod is kept after the agent finishes executing. It can have multiple options: never(), onFailure(), always(), or default(). If empty, it defaults to deleting the pod after the time specified by activeDeadlineSeconds.

  • activeDeadlineSeconds: If the podRetention parameter is set to never() or onFailure(), the pod will be automatically deleted after the time specified by this parameter.

  • idleMinutes: Allows the pod to remain active for reuse until the time specified by this value from the start of execution. The default value is in minutes.

  • showRawYaml: Enables or disables the output of the raw YAML file. The default value is true.

  • runAsUser: sets the user ID for running all containers in the pod.

  • runAsGroup: sets the group ID for running all containers in the pod.

  • hostNetwork: Uses the host network, similar to network=host in Docker.

  • workspaceVolume: The type of volume used for the workspace. It is the mount type of volume in Kubernetes and can be one of the following:

    • emptyDirWorkspaceVolume (default)
    • dynamicPVC()
    • hostPathWorkspaceVolume()
    • nfsWorkspaceVolume()
    • persistentVolumeClaimWorkspaceVolume()

Parameters of the Container Template Configuration:

  • name: The name of the container.

  • image: The image used by the container.

  • envVars: The environment variables applied to the container (supplementary to and overrides the envVars set at the pod level).

  • envVar: Same as above.

  • secretEnvVar: Same as above.

  • command: The command the container should execute.

  • args: The arguments to pass to the command.

  • ttyEnabled: Marks whether the container has a TTY enabled.

  • livenessProbe: The liveness probe for the container.

  • ports: The ports exposed by the container.

  • alwaysPullImage: The image pull policy.

  • runAsUser: The user ID used for running the container.

  • runAsGroup: The group ID used for running the container.

The above parameters are applicable to both script-based and declarative pipelines. For declarative syntax, you can also generate the corresponding PodTemplate syntax snippets using the snippet generator introduced earlier.

For more parameters related to Pod Template and Container Template, you can refer to the YAML definition of the pod resource object in Kubernetes. By default, the parameters supported in a pod can be used in the pipeline.

Now that we have understood the basic syntax, let’s deepen the understanding through some basic examples.

Syntax Examples #

Given the differences between script-based and declarative syntax, the examples will be presented separately.

Scripted Syntax #

The basic example is as follows:

podTemplate {
    node(POD_LABEL) {
        stage('Run shell') {
            sh 'echo hello world'
        }
    }
}

Explanation:

In the scripted syntax, the node() block is contained within the podTemplate method.

This example will start a Kubernetes Pod and output hello world.

node(POD_LABEL) : POD_LABEL is used to specify a label for the agent proxy (Kubernetes pod) of the pipeline, ensuring the uniqueness of the pod name. This label can be customized (specified through the label parameter of podTemplate), or it can be used as shown in the example, using POD_LABEL. This feature was added in version 1.17 of the kubernetes-plugin plugin and is used to automatically generate a pod label. The format of the label is ${JOB_NAME}-${BUILD_NUMBER}-hash_number.

From the above example, it can be seen that the PodTemplate does not define any pod or container configuration parameters. Without configuring these parameters, the pipeline will use the default PodTemplate configuration. After executing the above pipeline, the YAML definition of the Pod will be listed in the build log of Jenkins (as shown in the initial example). This includes the image version, Jenkins environment variables, etc. Since this is the default configuration, it does not provide much help in writing a pipeline for continuous delivery. If you are interested, you can execute the above example and the result will be displayed in the Jenkins console log.

For the basic example above, container parameters can be added to customize the startup parameters of the pod’s container, for example:

podTemplate(containers: [...]) {
    node(POD_LABEL) {
        stage('Run shell') {
            container('mycontainer') {
                sh 'echo hello world'
            }
        }
    }
}

In this example, the container parameter defines the image configuration template used to start the container. If the container parameter is not filled in, the default configuration parameters of the JNLP agent (container) are as follows:

containerTemplate(name: 'jnlp', image: 'jenkins/jnlp-slave:3.35-5-alpine', args: '${computer.jnlpmac} ${computer.name}')

Please note:

The ${computer.jnlpmac} and ${computer.name} parameters are required for running the JNLP agent (in this example, jnlp-slave:3.35-5-alpine) image. If one or both of these parameters are not included, it will cause the slave pod to fail to generate.

Another point to note is that for containers using JNLP agent images (which are launched and connected to the Jenkins master node through the slave agent package at startup), the container name must be jnlp. Otherwise, it may cause the pod to fail to generate. The reason for this will be explained in the next section.

The corresponding declarative syntax for the default JNLP agent (container) is as follows:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: jnlp
    image: 'jenkins/jnlp-slave:3.35-5-alpine'
    args: ['\$(JENKINS_SECRET)', '\$(JENKINS_NAME)']

From this example, you can understand the purpose of the two parameters mentioned in the scripted syntax, right? If you still don’t know, you can refer to docker-jnlp-slave for more information.

The default container configuration only includes the three parameters mentioned above: name, image, and args. In addition to these parameters, there are some commonly used parameters, such as ttyEnabled for allocating a pseudo-terminal, alwaysPullImage for setting the image pull policy, privileged for configuring whether to use privileged mode, and setting the container’s memory and CPU limits.

For more information on container-related parameter configurations, please refer to the following example:

containerTemplate(
    name: 'mariadb',
    image: 'mariadb:10.1',
    ttyEnabled: true,
    privileged: false,
    alwaysPullImage: false,
    workingDir: '/home/jenkins/agent',
    resourceRequestCpu: '50m',
    resourceLimitCpu: '100m',
    resourceRequestMemory: '100Mi',
    resourceLimitMemory: '200Mi',
    envVars: [
        envVar(key: 'MYSQL_ALLOW_EMPTY_PASSWORD', value: 'true'),
        secretEnvVar(key: 'MYSQL_PASSWORD', secretName: 'mysql-secret', secretKey: 'password'),
        ...
    ],
    ports: [portMapping(name: 'mysql', containerPort: 3306, hostPort: 3306)]
),

The parameters configuration related to podTemplate are as follows:

podTemplate(cloud: 'kubernetes', label: 'my-defined', containers: [
...
],
volumes: [
    emptyDirVolume(mountPath: '/etc/mount1', memory: false),
    secretVolume(mountPath: '/etc/mount2', secretName: 'my-secret'),
    configMapVolume(mountPath: '/etc/mount3', configMapName: 'my-config'),
    hostPathVolume(mountPath: '/etc/mount4', hostPath: '/mnt/my-mount'),
    nfsVolume(mountPath: '/etc/mount5', serverAddress: '127.0.0.1', serverPath: '/', readOnly: true),
    persistentVolumeClaim(mountPath: '/etc/mount6', claimName: 'myClaim', readOnly: true)
],
namespace: 'default',
serviceaccount: 'default',
imagePullSecrets: ['pull-secret'],  // Used to pull images from a private registry without authentication on the host machine
annotations: [
    podAnnotation(key: "my-key", value: "my-value")
    ...
])

When writing a pipeline script, if you don’t know how to define a particular argument in the podTemplate method, you can refer to the template example above or use the snippet generator introduced earlier to generate the syntax snippet through the Jenkins UI.

The configuration in the podTemplate method can be defined using the key:value format. It can also be defined using a YAML file, which is the default file extension for Kubernetes resource objects. For example:

podTemplate(yaml: """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: busybox
    image: busybox
    command:
    - cat
    tty: true
"""
) {
    node(POD_LABEL) {
        container('busybox') {
            sh "hostname"
        }
    }
}

In addition to using the yaml keyword, you can also use the yamlFile keyword to specify a YAML file. This file is typically located in the source code repository along with the Jenkinsfile. By fetching the Jenkinsfile and the file specified in this parameter from the source code repository, they will be downloaded and executed automatically. This approach requires changing the type of script defined in the pipeline job from the default Pipeline script to Pipeline script from SCM (as described in the previous section).

Running Multiple Containers

What if different code requires different build environments? Multiple containerTemplate can be defined in the containers parameter to meet this requirement. This allows running multiple containers within the same pod. In Kubernetes, containers within the same pod share hostname, network, volume, and other information, so running one or more containers within a pod doesn’t make much difference between them. For example:

podTemplate(label:'test', containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')
]) {

    node('test') {
        stage('Get a Maven project') {
            git 'https://github.com/jenkinsci/kubernetes-plugin.git'
            container('maven') {
                stage('Build a Maven project') {
                    sh 'echo build maven'
                }
            }
        }

        stage('Get a Golang project') {
            git url: 'https://github.com/hashicorp/terraform.git'
            container('golang') {
                stage('Build a Go project') {
                    sh 'echo build go'
                }
            }
        }

    }
}

In this example, multiple containers are defined, and the containers are used as the execution environment for different stages by specifying the container’s name.

Note:

This example defines multiple containers, and different stages use the specified containers by referencing their names.

Declarative Syntax #

Using the PodTemplate method in declarative syntax is similar to using it in scripted syntax. The main difference lies in how the agent is defined.

First, let’s take a look at a basic example of using PodTemplate in a declarative script:

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
metadata:
  labels:
    some-label: some-label-value
spec:
  containers:
  - name: maven
    image: maven:alpine
    command:
    - cat
    tty: true
  - name: busybox
    image: busybox
    command:
    - cat
    tty: true
"""
    }
  }
  stages {
    stage('Run maven') {
      steps {
        container('maven') {
          sh 'mvn -version'
        }
        container('busybox') {
          sh '/bin/busybox'
        }
      }
    }
  }
}

Explanation:

When using declarative syntax, the agent’s type must be one of “any, docker, dockerfile, kubernetes, label, none”. In this case, we are using the Kubernetes plugin, so the agent’s type is set to kubernetes.

The kubernetes block is used to define the Pod and container template configuration in YAML format. The container directive is used in the stage to specify the container to run and provide the environment for the pipeline execution.

This pod definition does not specify the namespace for the pod’s work. It is important to note that if the YAML definition does not specify a namespace, it will default to the namespace specified in the Jenkins system configuration when adding the Kubernetes cloud. If the namespace is not specified there either, it will default to the default namespace. If both are specified, it will default to the namespace defined in the pipeline script.

The way the containers are referenced in the stage is the same as the way containers are referenced in scripted pipelines.

For a more comprehensive definition of YAML, you can refer to the Kubernetes documentation for the pod resource object specification here.

In addition to using the yamlFile directive in scripted syntax, it can also be used in declarative syntax. Here’s an example:

pipeline {
  agent {
    kubernetes {
      yamlFile 'KubernetesPod.yaml'
    }
  }
  stages {
      ...
  }
}

Explanation:

In this example, the file KubernetesPod.yaml is specified, assuming the Jenkinsfile is in the same repository and at the same level. When configuring the job pipeline, the option to select Pipeline script from SCM is used. If the KubernetesPod.yaml file is not in the same level as the Jenkinsfile, the relative path needs to be specified.

Using Multiple Containers

By default, template definitions in declarative syntax do not inherit from parent templates. Therefore, even with a globally defined agent, you can still define agents separately in each stage without affecting each other.

In the following example, a global agent is defined, and an agent is also defined in the stage, and they do not interfere with each other:

pipeline {
  agent {
    kubernetes {
      label 'parent-pod'
      yaml """
spec:
  containers:
  - name: golang
    image: golang:1.6.3-alpine
    command:
    - cat 
    tty: true
"""
    }
  }
  stages {
    stage('Run maven') {
        agent {
            kubernetes {
                label 'nested-pod'
                yaml """
spec:
  containers:
  - name: maven
    image: maven:3.3.9-jdk-8-alpine
    command:
    - cat 
    tty: true
"""
            }
        }
      steps {
        container('maven') {
          sh 'mvn -version'
        }
      }
    }
    stage('run go'){
        steps {
        container('golang') {
          sh 'go --version'
        }
      }
    }
  }
}

These are some syntax examples used when dynamically generating Jenkins slave nodes with the kubernetes plugin. After learning the syntax and examples of integrating Jenkins with the Docker pipeline plugin in the previous sections, learning the content of this chapter should be relatively easier.

The next chapter will provide practical examples to demonstrate how to define PodTemplates and customize image mirrors in the Jenkins UI.