07 Application Deployment and Management Job DaemonSet #

在 Kubernetes 中，Job 和 DaemonSet 是两种常用的应用编排与管理的方式。

Job #

Job 是一种 Kubernetes 的控制器对象，用于控制和完成批处理任务。它确保一个或多个 Pod 成功运行到完成状态，并且可以按照用户指定的并行度进行扩展。

Job 通过创建 Pod 副本来完成任务。每个 Pod 副本将运行相同的容器，并且通过设置任务数量或完成情况来控制运行次数。当所有任务都完成时，Job 将标记为完成。

在 Job 完成时，可以通过查看 Pod 的日志或输出来检查任务的结果。

DaemonSet #

DaemonSet 是另一种 Kubernetes 的控制器对象，用于在集群中的每个节点上运行一个副本。它确保在集群中的每个节点上都运行相同的 Pod 副本。

与 ReplicaSet 和 Deployment 不同，DaemonSet 不是用于扩展应用程序的副本，而是用于在集群中的每个节点上运行必需的守护进程。

DaemonSet 通常用于进行一些集群级别的任务，例如日志收集、监控、网络代理等。

总结 #

Job 用于控制和完成批处理任务，确保任务的完成状态，在所有任务完成后标记为完成。而 DaemonSet 用于在集群中的每个节点上运行一个副本，确保在每个节点上都运行相同的 Pod。

这两种应用编排与管理的方式在不同情境下有着不同的用途和特点，可以根据实际需求来选择使用。

Job #

Source of Demand #

Background Issues of Job #

First, let’s take a look at the source of demand for Jobs. In Kubernetes, the smallest scheduling unit is a Pod, and we can run task processes directly through Pods. However, this approach raises several issues:

• How do we ensure that processes inside Pods are terminated correctly?
• How do we ensure that failed processes are retried?
• How do we manage multiple tasks with dependency relationships?
• How do we run tasks in parallel and manage the size of the task queue?

Job: A Controller for Managing Tasks #

Let’s see what functionalities Kubernetes provides for us with a Job:

• First, Kubernetes Job is a controller for managing tasks. It can create one or more Pods to specify the number of Pods and monitor whether they run or terminate successfully.
• We can set the restart policy and the number of retries based on the Pod’s status.
• We can ensure that the next task runs only after the previous task has completed based on the dependency relationship.
• We can also control the parallelism of tasks, ensuring the number of parallel runs during the execution of Pods and the overall completion size.

Use Case Interpretation #

Let’s take a look at how Job accomplishes the following application based on an example.

Job Syntax #

The above diagram represents the simplest yaml format of a Job. Here, we introduce a new kind called “Job,” which is actually a type in the job-controller. In the metadata, we specify the name of the Job. The spec.template section contains the specifications for the Pod.

The content inside is the same, with two additional points:

• The first is restartPolicy. In Job, we can set three types of retry strategies: Never, OnFailure, and Always. We can use Never when we want the Job to restart only when necessary, OnFailure when we want the Job to retry when it fails, or Always when we want the Job to restart in any case.
• Additionally, a Job cannot retry indefinitely, so we need a parameter to control the number of retries. This parameter is called backoffLimit, which determines how many times a Job can be retried.

So, in a Job, we mainly focus on the restartPolicy and the backoffLimit.

Job Status #

After creating a Job, we can use the command kubectl get jobs to check the current running status of the Job. The output will include the name of the Job, the number of completed Pods, and the duration of the execution.

The meaning of AGE refers to the time duration between the creation time of the Pod and the current time. This duration is mainly used to provide the history of the Pod and how long it has been created.

DURATION shows how long the actual service inside the Job has been running. This information is useful when optimizing performance. COMPLETIONS shows the total number of Pods and the number of completed Pods for the Job.

Viewing Pods #

Now, let’s take a look at Pods. Ultimately, a Job uses Pods as the execution unit. The Job we created earlier will generate a Pod named “pi,” which is responsible for calculating the area of a circle. The name of the pod is in the format “job−name−job−name−{random-suffix}”. Let’s look at the yaml format of the Pod below.

It is similar to a regular Pod, but it includes an additional field called ownerReferences. This field is used to declare which higher-level controller manages this Pod. In this case, the ownerReferences belongs to batch/v1, which means it is managed by the previous Job. This declaration specifies the controller of the Pod and allows us to retrieve the controller based on the Pod and see which Pods it owns based on the Job.

Running Jobs in Parallel #

Sometimes, we have a requirement to maximize parallelism during Job execution by running multiple Pods concurrently. However, since there is a limitation on the number of nodes, we may not want an excessive number of Pods running concurrently. To address this, we can set a maximum parallelism value, and the Job controller can handle this for us.

Here, two parameters are important: completions and parallelism.

• The first parameter is used to specify how many times this Pod queue will run. This may not be easy to understand, but you can think of it as the total number of runs permitted for this Job. For example, if we set it as 8, the task will be executed 8 times.
• The second parameter represents the number of parallel executions. The number of parallel executions is essentially the size of the buffer or queue in a pipeline. If we set it as 2, it means the Job must be executed 8 times, with 2 Pods running in parallel each time. In this way, there will be a total of 4 batches.

Viewing Jobs Running in Parallel #

Now, let’s take a look at the actual running effects of this Job after it has completed. In the above diagram, we can see the Job name, the total 8 Pods created, and the time taken for the execution, which is 2 minutes and 23 seconds.

Next, let’s look at the actual Pods. A total of 8 Pods were created, and each Pod is in the completed state. The AGE field shows the time elapsed. Looking from bottom to top, we can see the time duration for each group of Pods: 73s, 40s, 110s, and 2m26s. Two Pods have the same time in each group, where 40s represents the time when the last Pod is created, and 2m26s represents the time when the first Pod is created. In other words, two Pods are always created simultaneously, executed in parallel, completed, and then created again, running, and completing the process.

For example, we controlled the number of parallel executions using the second parameter. This shows the effect of the buffer or pipeline queue size.

CronJob Syntax #

Now let’s introduce another Job called CronJob, which can also be called a scheduled Job. CronJob is actually similar to Job in general, with the only difference being that it can be scheduled at a specific time. For example, it can be scheduled to run at a specific minute and hour. It is particularly suitable for performing cleanup tasks at night, as well as tasks that need to be executed every few minutes or hours. This is called a scheduled job.

Compared to a regular Job, a scheduled job has a few additional fields:

• schedule: This field is mainly used to set the time format. Its time format is the same as Linux’s crontime, so you can directly write it according to the format of Linux’s crontime. For example, “*/1” means to execute the Job every minute. The task of this Job is to print the approximate time and “Hello from the Kubernetes cluster”.
• startingDeadlineSeconds: This is the maximum time that the Job can wait when it runs. Sometimes, the Job may take a long time to start. In this case, if it exceeds the specified time, the CronJob will stop the Job.
• concurrencyPolicy: This refers to whether parallel execution is allowed. Parallel execution means that, for example, if I execute the Job every minute, but this Job may take a long time to run. If the second Job needs to run when the previous Job is still running, if this policy is set to true, it will execute every minute regardless of whether the previous Job has finished. If it is set to false, it will wait for the previous Job to finish before running the next one.
• JobsHistoryLimit: After each CronJob is completed, it will retain the execution history and viewing time of the previous Job. Of course, this cannot be infinite, so you need to set a limit on the number of historical records. Generally, you can set a default of 10 or 100, depending on the different clusters of each person, and then determine this value based on the number of clusters of each person.

Operation Demonstration #

Job Manifest File #

Let’s take a look at how to use a Job specifically.

Creating and Verifying a Job #

First, let’s take a look at job.yaml. This is a very simple task to calculate pi. Use kubectl create -f job.yaml to submit the job successfully. Let’s check kubectl get jobs, and we can see that this job is running. Checking kubectl get pods, we can see that the pod should have completed running. Next, let’s check the logs of this job and pod. We can see that the value of pi is printed in the image below.

Manifest File for Parallel Jobs #

Now let’s look at the second example:

Creating and Verifying Parallel Jobs #

This example is about creating parallel running jobs. After creating the jobs, we can see that there is a second parallel job.

Now there are two pods running, and we can see that it has been executing for almost 30 seconds.

After 30 seconds, it should start the second one.

The first batch of pods has been completed, and the second batch of pods is currently running, with two pods in each batch. This means that every 40 seconds or so, two pods will be executing in parallel. There will be a total of 4 batches and 8 pods executed in total, which is what we referred to as the parallel execution buffer queue.

After some time, when we check these pods again, we can see that the second batch has finished executing and the third batch is being created…

Configuration file for CronJob #

Next, let’s look at the third example - CronJob. CronJob is executed once every minute, with one job per execution.

Creating and running a CronJob #

As shown in the image below, the CronJob has already been created. You can use the command get cronjob to see that there is currently one CronJob. At this point, if we check the jobs, we need to wait for a little while since it runs once every minute.

At the same time, we can see that the previous job is still running. Its duration is about 2 minutes and 12 seconds, and its completion is 7/8, 6/8. Just now, we saw 7/8 to 8/8, which means our previous task completed its final step, and it ran with two jobs in parallel every time. Running two jobs at a time will be particularly convenient when running large workflows or tasks.

In the above figure, we can see that a job has suddenly appeared, named “hello-xxxx”. This job was created one minute after the previous CronJob was submitted. If we don’t interfere with it, it will create a job like this every minute in the future, unless we specify that it should not be allowed to run anymore.

In this case, CronJob is mainly used to perform cleanup tasks or execute scheduled tasks. For example, it is very effective for tasks related to Jenkins builds.

Architectural Design #

Job Management Mode #

Let’s take a look at the architectural design of a job. The Job Controller mainly creates the corresponding pods, and then tracks the status of the job, retrying or continuing creation according to our submitted configuration. As mentioned earlier, each pod will have its corresponding label to track the Job Controller it belongs to, and it will be configured to create pods in parallel or in series.

Job Controller #

The image above shows the main flow of a Job Controller. All jobs are controllers that watch the API Server. Every time we submit a Job yaml, it is sent through the API Server to ETCD. Then, the Job Controller registers several Handlers, and whenever there are operations such as add, update, or delete, it sends them to the controller through an in-memory message queue.

The Job Controller checks if there are any running pods. If not, it creates the pod by scaling up. If there are already pods running, or if there are more than the desired number, it scales down. If there are changes to the pod at this point, its status is updated promptly.

The Job Controller also checks whether the job is parallel or serial and creates the appropriate number of pods based on the configured parallelism or serialism. Finally, it updates the entire job’s status to the API Server, allowing us to see the final result.

DaemonSet #

Source of Demand #

Background Problem of DaemonSet #

Let’s introduce the second controller: DaemonSet. The same problem arises: what would happen if we didn’t have DaemonSet? Here are a few requirements:

• First, what should we do if we want every node to run the same pod?
• How do we fulfill the requirement of immediately detecting and deploying a pod to assist with initialization when a new node joins the cluster?
• If a node departs, how can we ensure that the corresponding pod is deleted?
• If a pod’s state is abnormal, how can we promptly monitor the abnormality on the node and take some monitoring or reporting actions? Which controller can help with these tasks?

DaemonSet: the Daemon Controller #

DaemonSet is also a default controller provided by Kubernetes. It actually serves as a daemon controller, which can help us achieve the following:

• First, it ensures that each node in the cluster runs a set of identical pods.
• It can automatically create corresponding pods based on the state of new nodes joining the cluster.
• It can delete corresponding pods when a node is removed.
• It tracks the state of each pod and promptly recovers the state when the pod becomes abnormal or crashes.

Use Case Analysis #

DaemonSet Syntax #

Let’s take a look at an example of a DaemonSet.yaml, which will be slightly longer.

First is kind: DaemonSet. If you have learned about deployment before, this yaml will be relatively simple to understand. For example, it will have matchLabels, which manage the corresponding pods that must match with DaemonSet.controller.labels, allowing it to find the managed pods based on the label.selector. The contents under spec.container are consistent.

Let’s use Fluentd as an example. The most common uses of DaemonSet are as follows:

• First, for storage, things like GlusterFS or Ceph require an agent-like process to run on each node. DaemonSet is well-suited to fulfill this requirement.
• Additionally, for log collection, such as Logstash or Fluentd, the same requirement applies, which requires an agent to run on each node. This way, we can easily collect the status and timely report the information from each node to the upper layer.
• Another example is the need for each node to run certain monitoring tasks. They also require each node to run the same tasks, such as Prometheus. This is also supported by DaemonSet.

Checking the Status of DaemonSet #

After creating a DaemonSet, we can use kubectl get DaemonSet (abbreviated as ds) to view the DaemonSet. We can see that the return value of DaemonSet is similar to that of deployment, including the total number of running pods and the number of ready pods. In this case, all that is created are pods.

There are several parameters: the number of pods required, the number of pods that have already been created, the number of ready pods, and all available pods that have passed health checks. NODE SELECTOR is also noteworthy because it is very useful in DaemonSet. Sometimes, we may want only certain nodes to run this pod instead of all nodes, so if some nodes are labeled, DaemonSet will only run on these nodes. For example, if we only want master nodes to run certain pods, or if we only want worker nodes to run certain pods, we can use this NODE SELECTOR.

Updating DaemonSet #

Actually, DaemonSet and Deployment are quite similar. They both have two update strategies: RollingUpdate and OnDelete.

• RollingUpdate is relatively easy to understand. It updates the pods one by one. It first updates the first pod, then removes the old pod, and once it passes the health check, it moves on to the second pod. This ensures a smooth upgrade without interruption to the business.
• OnDelete is also a good update strategy. After updating the template, the pods will not change automatically. We need to manually control them. If we delete the pod corresponding to a specific node, it will be rebuilt. If we don’t delete it, it won’t be rebuilt. This is useful for special requirements that need manual control.

Operation Demonstration #

Deployment of DaemonSet #

Here is an example. Suppose we make some changes to the image of the DaemonSet, and then observe its state as it updates one by one.

The above image is the yaml of the DaemonSet we just saw, with some additional resource limitations, which do not affect the functionality.

Creation and Running Verification of DaemonSet #

Now let’s create the DaemonSet and check its status. The following image shows the state of the DaemonSet printed in the “ready” section.

From the image above, we can see that a total of 4 pods have been created. Why 4 pods? Because there are only 4 nodes, so there will be one pod running on each node.

Update of DaemonSet #

Now, let’s update the DaemonSet. After executing kubectl apply -f, the DaemonSet will be updated. Let’s check the update status of the DaemonSet.

In the image above, you can see that the DaemonSet is using the default RollingUpdate. The previous version was labeled as 0-4, and now it is labeled as 1-4, which means it is updating the first pod, then the second pod, and so on. This is RollingUpdate. RollingUpdate can achieve fully automated updates without human intervention. Instead, it automatically updates one by one, making the update process smoother. This can be beneficial for on-site releases or other operations.

At the end of the image, you can see that the entire DaemonSet has finished RollingUpdate.

Architectural Design #

DaemonSet Management Mode #

Next, let’s take a look at the architectural design of DaemonSet. DaemonSet is still a controller, and its actual business unit is also the Pod. DaemonSet is similar to Job controller in that it watches the status of the API Server through the controller and adds pods in a timely manner. The only difference is that it monitors the state of the nodes. When a node joins or leaves, it will create a corresponding pod on the node, and then select the corresponding node based on the affinity or label configuration.

DaemonSet Controller #

Finally, let’s take a look at the DaemonSet controller. DaemonSet and Job controller are quite similar: they both need to watch the state of the API Server. The only difference between DaemonSet and Job controller is that the DaemonSet controller needs to watch the state of the nodes. However, the state of the nodes is still passed to ETCD via the API Server.

When the state of a node changes, it is sent through a memory message queue, and then the DaemonSet controller watches this state to see if there is a corresponding pod on each node. If not, it creates one. Of course, it also compares the versions. And, as mentioned earlier, it checks whether to perform RollingUpdate. When deleting a pod in OnDelete, it also checks whether to update or create the corresponding pod.

Finally, when all the updates are completed, it updates the entire status of the DaemonSet to the API Server, completing the final update process.

Summary of this section #

• Basic operations and concept analysis of Jobs & CronJobs: This section provides a detailed introduction to the concepts of Jobs and CronJobs. Two practical examples are given to demonstrate the usage of Jobs and CronJobs, and various features of Jobs and CronJobs are demonstrated in detail.
• Basic operations and concept analysis of DaemonSets: By analogizing to the Deployment controller, we understand the workflow and method of the DaemonSet controller. We also learn about the concept of rolling updates and its corresponding operation methods through understanding the updates of DaemonSets.