19 Daemonset the Loyal and Reliable Watchdog

19 Daemonset The Loyal and Reliable Watchdog #

Hello, I am Chrono.

In the last lesson, we learned about a new API object in Kubernetes called Deployment. It represents an online business and can manage multiple replicas of pods, keeping applications always online and allowing for easy scaling up or down.

Although Deployment is very useful, it does not fully solve all the challenges of deploying applications in operations. This is because online business scenarios are too diverse and complex. The features of Deployment only cover a portion of these scenarios and cannot meet the requirements of other use cases.

Today, let’s take a look at another type of API object that represents online business: DaemonSet. It runs a pod on every node in the Kubernetes cluster, just like a “daemon” in a Linux system.

Why do we need DaemonSet? #

To understand why Kubernetes introduced the DaemonSet object, we need to understand the limitations of Deployment.

Let’s first briefly review Deployment. It can create multiple instances of Pods and ensure that these Pods are running properly, thus keeping the application always available.

However, Deployment does not care about which nodes in the cluster these Pods will run on. From its perspective, the running environment and functionality of the Pods are irrelevant. As long as there are sufficient number of Pods, the application should work fine.

This assumption is not a problem for most businesses. For example, Nginx, WordPress, and MySQL don’t need to know the details of the cluster or nodes. They work the same as long as the environment variables and storage volumes are configured properly, regardless of where they “run”.

But there are some special cases where businesses are closely tied to the host system and must be attached to nodes to generate value. For example:

  • Network applications (such as kube-proxy) need to run one Pod per node, otherwise the node cannot join the Kubernetes network.
  • Monitoring applications (such as Prometheus) require one Pod per node to monitor the status of the node and report information in real-time.
  • Log applications (such as Fluentd) need to run one Pod per node to collect logs generated by container runtimes.
  • Security applications also require one Pod per node to perform security audits, intrusion detection, vulnerability scanning, and so on.

Using Deployment to deploy these types of applications is not suitable because Deployment manages a fixed number of Pods and these Pods may “migrate” within the cluster. However, the actual requirement is to run Pods on each node in the cluster, meaning that the number of Pods should be synchronized with the number of nodes.

Therefore, Kubernetes defines a new API object called DaemonSet. It is similar to Deployment in form as it manages and controls Pods, but the management and scheduling strategy is different. The goal of DaemonSet is to run and only run one Pod on each node in the cluster, just like having a “watchdog” assigned to each node, faithfully “guarding” the node. That’s where the name DaemonSet comes from.

How to Use YAML to Describe DaemonSet #

Both DaemonSet and Deployment belong to the “apps” group since they are both used for online businesses. You can use the command kubectl api-resources to find out that the abbreviated name for DaemonSet is ds. The YAML file header information should be as follows:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: xxx-ds

However, it is strange that kubectl does not provide a function to automatically create a DaemonSet YAML template. This means that we cannot directly create a DaemonSet object using the command kubectl create.

Image

This drawback indeed causes some trouble for us when using DaemonSet. After all, it is quite tedious to use kubectl explain to check the fields one by one and then write the YAML file.

However, even though Kubernetes does not provide us with an opportunity to generate template files, we can still “copy” them ourselves. You can find a DaemonSet YAML example on the official Kubernetes website (https://kubernetes.io/zh/docs/concepts/workloads/controllers/daemonset/), copy it and remove the unnecessary parts. Then you can create your own template file, which would look something like this:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: redis-ds
  labels:
    app: redis-ds

spec:
  selector:
    matchLabels:
      name: redis-ds

  template:
    metadata:
      labels:
        name: redis-ds
    spec:
      containers:
      - image: redis:5-alpine
        name: redis
        ports:
        - containerPort: 6379

The name of this DaemonSet object is redis-ds and it uses the image redis:5-alpine, which is a popular NoSQL database (you may be familiar with it).

If you compare this YAML with the Deployment object we discussed in the previous lesson, you will notice that the kind and metadata sections are specific to each object and therefore different. However, the spec section, which is the important part, is almost identical to the Deployment object. Additionally, the DaemonSet object has a selector field in the spec section, which matches the labels of the Pods in the template.

If you pay close attention, you will notice that the DaemonSet does not have a replicas field in the spec section. This is a key difference from Deployment, which means that the DaemonSet will not create multiple replica Pods in the cluster but rather only one Pod instance on each node.

Therefore, in a way, you can consider DaemonSet as a special case of Deployment because everything else is similar. I have also created an image to illustrate the differences between DaemonSet and Deployment:

Image

Now that you understand these differences, we can create a YAML template for DaemonSet using a workaround. You only need to first create a Deployment object using kubectl create and then change the kind to DaemonSet and remove the spec.replicas. For example:

export out="--dry-run=client -o yaml"

# Change "kind" to DaemonSet
kubectl create deploy redis-ds --image=redis:5-alpine $out

How to use DaemonSet in Kubernetes #

Now, let’s execute the kubectl apply command to send the YAML to Kubernetes and create the DaemonSet object. Then, use kubectl get to check the status of the object:

Image

Looking at this screenshot, even though we haven’t specified the number of Pods to run in the DaemonSet, it will automatically search for nodes in the cluster and create Pods on those nodes. In our test environment, there is one Master and one Worker, and the Master node doesn’t run applications by default, so the DaemonSet only generated one Pod, running on the “worker” node.

Now, pause for a moment, do you notice anything strange here?

According to the intention of the DaemonSet, there should be one Pod instance running on each node. However, the Master node is excluded, which is not in line with our original plan.

Clearly, the DaemonSet has not fulfilled its “watchdog” responsibility, and its design conflicts with the working mechanism of the Kubernetes cluster. Is there a solution?

Of course, Kubernetes has already thought about this. In order to address the “scheduling” and “eviction” issues of Pods on certain nodes, it defines two new concepts: taint and toleration.

What are Taint and Toleration? #

“Taint” is a property of Kubernetes nodes that acts as a “label” for the node. However, to avoid confusion with the existing labels field, it is referred to as “taint”.

In contrast to “taint”, “toleration” refers to whether a Pod can “tolerate” a taint.

To understand these concepts, let’s put them together. Nodes in a cluster can vary, with some nodes being “pure” and without any “taint”, while others may have acquired a “taint” due to specific reasons. Pods also have their own preferences, with some being highly sensitive to cleanliness and unable to tolerate any “taint”, while others are more lenient and can tolerate some minor “taint”.

In this sense, “taint” and “toleration” can be compared to a dating process. Pods act as the picky “party A”, while the various nodes in the cluster act as the “party B”. Based on the Pod’s “tolerance” for “taint”, it selects a suitable target, just as it may require “no smoking, no drinking”, but can tolerate “no car, no house” and ultimately decides which node to reside on.

When creating a cluster, Kubernetes automatically adds some “taints” to the nodes, which facilitate Pod scheduling and deployment. You can use kubectl describe node to view the status of the Master and Worker nodes. For example:

kubectl describe node master

Name:     master
Roles:    control-plane,master
...
Taints:   node-role.kubernetes.io/master:NoSchedule
...

kubectl describe node worker

Name:     worker
Roles:    <none>
...
Taints:   <none>
...

As you can see, the Master node has a default taint called node-role.kubernetes.io/master, with an effect of NoSchedule, which means that this taint prevents Pod scheduling on the node. However, the Worker node does not have any taints specified.

This difference in Pod scheduling strategy between the Master and Worker is because Pods generally cannot tolerate any “taint”. Therefore, the Master node with the taint attribute would be unable to host any Pods.

Now that you understand the concepts of “taint” and “toleration”, you will know how to make a DaemonSet run on the Master node (or any other node). There are two methods to achieve this.

The first method is to remove the taint from the Master node, making it as “pure” as a Worker node. In this case, the DaemonSet no longer needs to differentiate between Master and Worker.

To remove the “NoSchedule” effect from the Master node, you can use the kubectl taint command, specifying the node name, taint name, and the effect of the taint, followed by a - to remove it.

For example, to remove the “NoSchedule” effect from the Master node, you can use the following command:

kubectl taint node master node-role.kubernetes.io/master:NoSchedule-

Because the DaemonSet continuously monitors the status of the cluster nodes, once the command is executed, the Master node no longer has the “taint”. Therefore, the DaemonSet immediately detects the change and creates a “master” Pod on the Master node. You can use kubectl get to view this change:

kubectl get ...

However, this method modifies the status of the node, which can have a significant impact as it may lead to many Pods running on that node. Therefore, an alternative method is to keep the “taint” on the node and add “tolerations” to the Pods that require it. This allows the Pods to run on specific nodes, achieving more precise scheduling.

This leads us to the second method, which involves adding the tolerations field to the Pod, allowing it to “tolerate” certain “taints” and enabling it to run on any node.

tolerations is an array that contains multiple “taints” that are “tolerated”. It specifies the name and effect of the “taint”. It is important to note that the operator field determines how the “taint” is matched. The Exists operator is commonly used, indicating the presence of the specified name and effect of the “taint”.

If we want Pods in a DaemonSet to run on the Master node, we need to specify a tolerations field that can tolerate the node-role.kubernetes.io/master:NoSchedule taint:

tolerations:
- key: node-role.kubernetes.io/master
  effect: NoSchedule
  operator: Exists

First, we use the kubectl taint command to add the “taint” to the Master node:

kubectl taint node master node-role.kubernetes.io/master:NoSchedule

Then, we redeploy the DaemonSet with the added “tolerations”:

kubectl apply -f ds.yml

Now, you will see that the DaemonSet still has two Pods, running on the Master and Worker nodes respectively, just like in the first method.

It is important to note that “toleration” is not exclusive to DaemonSets. It is a property of Pods, which means that once you understand the concepts of “taint” and “toleration”, you can also add tolerations to Pods managed by Jobs/CronJobs and Deployments, allowing for more flexible scheduling.

As for different types of “taints” and their effects, I won’t go into detail here. The Kubernetes official documentation (https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) provides a clear list. Once you understand the principles of how they work, I believe you will find it easy to learn on your own.

What is a StaticPod #

DaemonSet is the most common way to run node-specific Pods in Kubernetes. However, it is not the only way. Kubernetes also supports another method called “StaticPod” for application deployment.

A “StaticPod” is quite special. It is not managed by the Kubernetes system, and it is not associated with apiserver or scheduler, hence the term “static”.

However, since it is still a Pod, it runs on a container runtime and is described by a YAML file. The only component able to manage it is the kubelet running on each node.

By default, the YAML files for StaticPods are stored in the /etc/kubernetes/manifests directory on the node, which is a dedicated directory in Kubernetes.

The following screenshot shows the contents of the directory on the Master node:

picture

As you can see, the four core Kubernetes components: apiserver, etcd, scheduler, and controller-manager, are all in the form of StaticPods. This is why they can start before the Kubernetes cluster.

If you have special requirements that cannot be fulfilled by a DaemonSet, you may consider using a StaticPod. Write a YAML file and place it in this directory, and the kubelet on the node will periodically check the directory for changes and create or remove the StaticPod accordingly.

Summary #

Alright, today we learned about another way to deploy applications in Kubernetes: DaemonSets. They are similar to Deployments, with the only difference being the scheduling strategy for Pods. They are suitable for running “daemon processes” on each node in the system.

Let’s summarize today’s content:

  1. The goal of a DaemonSet is to deploy a unique Pod on every node in the cluster, commonly used for monitoring, logging, and other similar tasks.
  2. The YAML description of a DaemonSet is very similar to a Deployment, except it does not have the replicas field.
  3. “Taints” and “tolerations” are two important concepts related to DaemonSets. They belong to Node and Pod respectively, and together determine the scheduling strategy for Pods.
  4. Static Pods can achieve the same effect as DaemonSets, but they are not managed by Kubernetes and must be manually deployed on nodes. They should be used with caution.

Homework #

Now it’s time for homework. Here are two questions for you to think about:

  1. What are the differences between DaemonSet and Deployment in terms of usage? What scenarios are they suitable for?
  2. How do you think the concepts of “taints” and “tolerations” should be used effectively in Kubernetes?

Feel free to leave a comment and share your thoughts to participate in the discussion with your classmates. See you in the next class.