05 Application Deployment and Management Core Principles #

Resource Metadata #

1. Kubernetes Resource Objects #

First of all, let’s review the composition of Kubernetes resource objects: they mainly consist of two parts, Spec and Status. The Spec part is used to describe the desired state, while the Status part is used to describe the observed state.

Today, we will introduce another part of K8s, the metadata part. This part mainly includes the labels used to identify resources, the annotations used to describe resources, and the OwnerReference used to describe the relationships between multiple resources. These metadata play a very important role in the operation of K8s. They will be repeatedly discussed in subsequent courses.

2. Labels #

The first metadata, and also the most important one, is the resource label. The resource label is a type of key-value metadata with identifying characteristics. Here are some common labels.

The first three labels are applied to Pod objects, respectively indicating the corresponding application environment, deployment maturity, and application version. From the example of application label, we can see that the name of the label includes a domain name prefix, which is used to describe the system and tool for labeling. The last label is applied to the Node object, and it also adds the version identifier “beta” to the domain name prefix.

Labels are mainly used for filtering and grouping resources. Similar to SQL query select, you can use Label to query related resources.

3. Selector #

The most common Selector is the equality-based Selector. Let’s take a simple example now.

Assume that there are four Pods in the system, each of which has labels that identify the system hierarchy and environment. By using the Selector Tie=front, we can match the Pods shown in the left column. An equality-based Selector can also include multiple equality conditions, and these conditions are logically “AND” relationships.

In the previous example, the Selector Tie=front,Env=dev allows us to filter out all Pods that have Tie=front and Env=dev, which is the Pod in the top-left corner of the figure below. Another type of Selector is the set-based Selector. In the example, the Selector filters out all Pods with the environment set to “test” or “gray”.

In addition to the set operation of “in”, there is also the set operation of “notin”. For example, Tie notin (front,back) will filter out all Pods that are not front and back. Moreover, you can also filter based on the existence of a certain label, such as Selector release, which filters out all Pods with the release label. Both set-based and equality-based Selectors can be connected by “,” to represent a logical “AND” relationship.

4. Annotations #

Another important metadata is: annotations. They are generally used by systems or tools to store non-identifying information about resources. They can be used to extend the description of the spec/status of resources. Here are some examples of annotations:

In the first example, it stores the certificate ID of Alibaba Cloud’s load balancer. We can see that annotations can also have domain name prefixes, and version information can be included in the annotations. The second annotation stores the configuration information of the nginx ingress layer. We can see that annotations can include special characters like “,” which cannot appear in labels. The third annotations are generally seen in the resources after the operation of the kubectl apply command line. The annotation value is a structured data, actually a JSON string, which marks the description of the JSON of the resource from the last kubectl operation.

5. OwnerReference #

We mentioned the last metadata called OwnerReference before. The so-called owner generally refers to collection-type resources, such as Pod collections, which include ReplicaSet and StatefulSet, which will be discussed in subsequent courses.

The controller of a collection-type resource will create the corresponding owning resource. For example, the ReplicaSet controller will create Pods during operation, and the OwnerReference of the created Pods points to the ReplicaSet that created them. OwnerReference allows users to easily find the object that created a resource, and it can also be used to achieve cascading deletion effects.

Operation Demonstration #

Here, we use the kubectl command to connect to a K8s cluster that has already been created in ACK, and then demonstrate how to view and modify the metadata in K8s objects, mainly including labels, annotations, and corresponding owner references for Pods.

First, let’s take a look at the current configuration in the cluster:

To view Pods, there are currently no Pods created;

kubectl get pods

Then use a pre-prepared YAML file to create a Pod;

kubectl apply -f pod1.yaml
kubectl apply -f pod2.yaml

Now let’s take a look at the labels attached to the Pods. We can use the --show-labels option to see that these two Pods have been labeled with a deployment environment and hierarchy;

kubectl get pods --show-labels

We can also use another method to view specific resource information. First, let’s view the information of the first Pod, nginx1, using the -o yaml option to output in YAML format, which includes a labels field in the metadata of this Pod, with two labels inside;

kubectl get pods nginx1 -o yaml | less

Now, let’s think about how to modify the existing labels of a Pod. Let’s change its deployment environment from development to testing, and then specify the name of the Pod, and add its value as test, to see if it is successful. Here, an error is reported, indicating that this label already has a value;

kubectl label pods nginx1 env=test

If you want to overwrite it, you need to add an additional --overwrite option. After adding it, we should be able to see that this label has been successfully added;

kubectl label pods nginx1 env=test --overwrite

Let’s take a look at the current label settings in the cluster again. We can see that nginx1 has indeed added a deployment environment test label;

kubectl get pods --show-labels

To remove a label from a Pod, the operation is similar to labeling, but the equal sign is replaced by a hyphen following the label name when removing a label, i.e., using the format key- to remove a label;

kubectl label pods nginx tie-

We can see that the label removal is successful;

kubectl get pods --show-labels

Now, let’s take a look at the configured label values and see that the Pod nginx1 no longer has the tie=front label. Once this label is added to the Pod, let’s see how to use label selectors to match? Label selectors are specified using the -l option. When specifying, let’s start by trying to filter by an equality type label, so we specify it as the Pod with the deployment environment equal to “test”, and we can see that one Pod is filtered out;

kubectl get pods --show-labels -l env=test

If there are multiple equal conditions that need to be specified, they are effectively an “and” relationship. For example, if env is equal to “dev”, we won’t get any Pods;

kubectl get pods --show-labels -l env=test,env=dev

Then, let’s say if env=dev, but tie=front, we can match the second Pod, which is nginx2;

kubectl get pods --show-labels -l env=dev,tie=front

We can also try using a set-based label selector to filter. This time, we still want to match all Pods with the deployment environment of “test” or “dev”, so we add quotes here and specify all the deployment environments in the brackets. This time, we can filter out both created Pods;

kubectl get pods --show-labels -l 'env in (dev,test)'

Next, let’s try adding an annotation to a Pod. The operation for annotation is similar to labeling, but instead of the label command, we use the annotate command. Then, specify the type and corresponding name as usual. Instead of adding the k:v format for labels, we add the k:v format for annotations. Here, we can specify an arbitrary string, such as adding a space or a comma;

kubectl annotate pods nginx1 my-annotate='my annotate,ok'

Then, let’s take a look at some metadata of this Pod. Here, we can see that the metadata of this Pod contains the annotations, which has an annotation called my-annotate;

kubectl get pods nging1 -o yaml | less

We can also see that when using kubectl apply, the kubectl tool adds an annotation, which is a JSON string.

Then, let’s demonstrate how the OwnerReference of a Pod is generated. The original Pods were created directly by creating Pod resources. This time, let’s create Pods through a ReplicaSet object. First, create a ReplicaSet object and take a look at the details;

kubectl apply -f rs.yaml
kubectl get replicasets nginx-replicasets -o yaml |less

1. We can focus on the spec inside the ReplicaSet, which mentions that it will create two Pods, and the selector matches the label where the deployment environment is product. So let’s take a look at the Pods in the cluster now;

kubectl get pods

We will find that there are two more Pods. Check these two Pods carefully, and you will see that the Pods created by the ReplicaSet have a feature, which is that they have an OwnerReference, and the OwnerReference points to a ReplicaSet named nginx-replicasets;

kubectl get pods nginx-replicasets-rhd68 -o yaml | less

Controller Pattern #

1. Control Loop #

The core concept of the controller pattern is the control loop. The control loop consists of three logical components: the controller, the controlled system, and the sensors that can observe the system.

These components are all logical, and external changes to resources are made by modifying the resource spec. The controller compares the resource spec and status to calculate a diff. This diff is used to determine what control operation should be performed on the system. The control operation creates new output for the system, which is reported by the sensors as the resource status. Each component of the controller operates independently and continuously moves the system towards the desired state represented by the spec.

2. Sensors #

The logical sensors in the control loop are mainly composed of the Reflector, Informer, and Indexer components.

The Reflector retrieves resource data from the List and Watch K8s server. The List operation is used to perform a full update of system resources when the controller restarts or when the Watch operation is interrupted. The Watch operation performs incremental updates between multiple List operations. After the Reflector obtains new resource data, it puts a Delta record in the Delta queue, which includes the resource object information and the event type of the resource object. The Delta queue ensures that each object has only one record, avoiding duplicate records when the Reflector performs List and Watch operations again.

The Informer component continuously pops Delta records from the Delta queue, then passes the resource objects to the indexer so that the indexer can store the resource in a cache. By default, the cache is indexed using the resource’s namespace and can be shared by the Controller Manager or multiple controllers. Then, the event is passed to the callback function for processing.

The controller component in the control loop mainly consists of event handling functions and workers. The event handling functions monitor events related to the addition, update, and deletion of resources and decide whether they need to be processed according to the logic of the controller. For events that need to be processed, the event is associated with the namespace and name of the resource and put into a work queue. The work is then handled by a Worker from the worker pool. The work queue deduplicates the stored objects to avoid multiple workers processing the same resource.

When a Worker processes a resource object, it usually needs to retrieve the latest resource data using the resource name. This data is used to create or update the resource object or call other external services. If the Worker fails to process the resource, it usually puts the resource name back into the work queue for future retries.

3. Control Loop Example - Scaling #

Let’s take a simple example to illustrate how the control loop works.

A ReplicaSet is a resource used to describe the scaling behavior of stateless applications. The ReplicaSet controller maintains the desired number of replicas for the application by watching the ReplicaSet resource. Consider the scenario where the replicas value in ReplicaSet rsA is changed from 2 to 3.

First, the Reflector watches for changes in ReplicaSet and Pod resources. We’ll explain why we also watch the Pod resource later. When a change is detected in the ReplicaSet, an update record with rsA as the object and the type of update is put into the Delta queue.

The Informer updates the new ReplicaSet to the cache, indexing it with Namespace nsA. Additionally, it calls the Update callback function. When the ReplicaSet controller detects a change in the ReplicaSet, it puts the string nsA/rsA into the work queue. A Worker from the work queue retrieves the nsA/rsA key and fetches the latest ReplicaSet data from the cache.

The Worker compares the values in the ReplicaSet spec and status and determines that scaling is required. It then creates a Pod, with the OwnerReference pointing to ReplicaSet rsA.

Subsequently, when the Reflector watches the Pod and detects a new Pod event, an additional Delta record of type “Add” is added to the Delta queue. The record is stored in the cache using the Indexer, and the Add callback function of the ReplicaSet controller is invoked. The Add callback function identifies the corresponding ReplicaSet by checking the Pod’s OwnerReferences and puts the namespace and name of the ReplicaSet in the work queue.

When the ReplicaSet’s Worker retrieves the new work item, it fetches the latest ReplicaSet record from the cache and obtains all Pods created by the ReplicaSet. Since the ReplicaSet’s status is not up to date, meaning the number of created Pods is not the latest, the Worker updates the ReplicaSet status to align the spec and status.

Summary of the Controller Pattern #

1. Two API Design Methods #

The Kubernetes controller pattern relies on a declarative API. Another common API type is an imperative API. Why does Kubernetes use a declarative API instead of an imperative API to design the entire controller?

First, let’s compare the differences in interaction between the two APIs. In everyday life, a common example of imperative interaction is the way parents communicate with their children. This is because children often lack a sense of goals and cannot understand what parents expect. Parents often use commands to teach children specific actions, such as commands like “eat” or “sleep”. In the container orchestration system, an imperative API is executed through explicit operations on the system.

On the other hand, a common example of declarative interaction is the way a boss communicates with their employees. A boss generally does not give very specific decisions to their employees and may not even understand the tasks themselves as well as the employees. Therefore, bosses use quantifiable business goals to leverage the employees’ subjective initiative. For example, a boss may require that a certain product’s market share reach 80%, without specifying the specific operational details to achieve this market share.

Similarly, in the container orchestration system, we can keep the number of application instance replicas at 3 without explicitly scaling or deleting existing pods to maintain the desired number of replicas.

2. Issues with Imperative API #

After understanding the differences between the two interaction APIs, let’s analyze the problems with an imperative API.

• The biggest problem with the imperative API is error handling.

In large-scale distributed systems, errors are ubiquitous. Once a command does not receive a response, the caller can only attempt to recover from the error by retrying repeatedly. However, blind retries can lead to even greater problems.

For example, suppose the original command has already been executed in the background, but after retrying, an additional retry command is executed. To avoid the problems caused by retries, the system often needs to record the commands that need to be executed before executing the commands. In scenarios such as restarts, the system needs to redo the pending commands. During the execution, complex logic regarding the order and coverage of multiple commands needs to be considered.

• In fact, many imperative interaction systems often have a monitoring system to correct data inconsistencies caused by scenarios such as command processing timeouts and retries.

However, because the monitoring logic is different from the daily operational logic, it often lacks sufficient test coverage and rigorous error handling, posing a significant operational risk. Therefore, many monitoring systems are manually triggered.

• Finally, imperative APIs are also prone to problems when handling multiple concurrent accesses.

If multiple parties concurrently request the operation on a resource, and once an error occurs in one of the operations, it is difficult to determine which operation took effect last and cannot be guaranteed. Many imperative systems often lock the system before the operation to ensure the predictability of the final effective behavior of the entire system. However, the locking behavior will reduce the operational efficiency of the entire system.

• On the other hand, in a declarative API system, the current and final states of the system are naturally recorded.

No additional operational data is required. In addition, due to the idempotence of the state, operations can be repeated at any time. In the declarative system’s operating mode, normal operations are actually an inspection of the resource state, eliminating the need for the development of additional monitoring systems. The operational logic of the system can also be tested and refined in daily operations, ensuring the stability of the entire operation.

Finally, because the final state of the resource is clear, multiple modifications to the state can be merged. Locking is not required, and multiple concurrent accesses are supported.

3. Summary of the Controller Pattern #

Finally, let’s summarize:

1. The controller pattern adopted by Kubernetes is driven by a declarative API. More precisely, it is driven by modifications to Kubernetes resource objects.
2. After Kubernetes resources are established, controllers that focus on these resources are established. These controllers asynchronously drive the system towards the set final state.
3. These controllers operate autonomously, enabling system automation and unattended operation.
4. Because Kubernetes controllers and resources can be customized, the controller pattern can be easily extended. In particular, for stateful applications, we often use custom resources and controllers to automate operations. This is also the scenario that will be introduced in the following section, the operator.

Summary #

The main content of this lesson ends here. Let’s summarize briefly:

• The metadata section in Kubernetes resource objects mainly includes labels to identify resources, annotations to describe resources, and OwnerReferences to describe the relationship between multiple resources. These metadata play a crucial role in the operation of Kubernetes.
• The core concept in the control loop in the control pattern.
• There are two API design methods: declarative API and imperative API. The controller pattern adopted by Kubernetes is driven by the declarative API.