06 Application Deployment and Management #

In this module, we will explore the concept of application orchestration and management. This involves the process of organizing and coordinating various components of an application to ensure its smooth operation.

We will cover topics such as:

• Containerization and container management
• Orchestration frameworks like Kubernetes
• Deployment strategies
• Monitoring and logging
• Scaling and load balancing

By the end of this module, you will have a good understanding of how to effectively orchestrate and manage applications, ensuring their reliability, scalability, and performance.

Source of requirements #

Background issue #

First, let’s look at the background issue. As shown in the figure below: if we directly manage all the pods in the cluster, the pods of applications A, B, and C are actually scattered throughout the cluster.

Now, we have the following issues:

• First, how to ensure the number of available pods in the cluster? In other words, if there are some host failures or network problems, how can we ensure the availability of four pods for application A?
• How to update the image version for all pods? Do we need to recreate a new version of a pod for a certain pod?
• During the update process, how to ensure service availability?
• And during the update process, if a problem is found, how to quickly roll back to the previous version?

Deployment: Controller for managing deployment releases #

This introduces the theme of our today’s lesson: Deployment, a controller for managing deployment releases.

As we can see, we use Deployment to plan applications A, B, and C into different deployments, and each deployment actually manages a group of identical application pods. We consider this group of pods as the same replica. So, what can Deployment do for us?

First, Deployment defines an expected number of pods, for example, for application A, we expect four pods. In this way, the controller will continuously maintain the number of pods to match the expected quantity. When we encounter network problems or host issues with the pods, the controller can help us recover, meaning it will scale out new pods to ensure the availability of pods matches the expected quantity.

It can configure how pods are released, meaning the controller will update the pods according to the given user-defined strategies. Additionally, during the update process, the controller can set the acceptable range of unavailable pods.

If any issues occur during the update process, it supports “one-click” rollback, which means that you can update all the pods under the Deployment to a certain old version with just one command or one line modification.

Use Case Interpretation #

Deployment Syntax #

Let’s interpret how to operate a Deployment using a simple use case.

The image above shows a YAML file of the simplest Deployment.

“apiVersion: apps/v1” indicates that the current group of the Deployment is “apps” and the version is “v1”. “metadata” represents the metadata of the Deployment, which includes labels, selectors, and the Pod.image mentioned in previous lessons.

As a Kubernetes resource, Deployment has its own metadata. In this case, the Deployment name is “nginx.Deployment”. In the Deployment.spec, there is a core field, “replicas”, which defines the desired number of Pods as three. The “selector” is a Pod selector, which means that all the scaled Pods must have labels that match the labels defined in “selector” under the “image.labels”, which is “app.nginx”.

As described in the Pod template above, the template contains two parts:

• One part is the expected metadata of the Pod, including labels that match the “selector.matchLabels”.
• The second part is the Pod.spec contained in the template. Here, the Pod.spec used when the Deployment creates Pods is defined. In this case, it defines a “container.nginx” with the image version “nginx:1.7.9”.

The new knowledge points encountered here are:

• The first is “replicas”, which refers to the expected or final number of Pods in the Deployment.
• The second is the “template”, which is a template related to Pods.

View Deployment Status #

When we create a Deployment, we can use the command “kubectl get deployment” to see the overall status of the Deployment. As shown in the figure below:

In the image above, we can see the following:

• DESIRED: The desired number of Pods is 3.
• CURRENT: The current actual number of Pods is 3.
• UP-TO-DATE: The number of Pods that have reached the latest expected version.
• AVAILABLE: This indicates the number of Pods that are actually available during runtime. It should be noted that “AVAILABLE” does not simply mean “Ready” state. It includes Pods that have been available for a certain period of time.
• AGE: The duration of the deployment. In the example above, the Deployment has been running for 80 minutes.

View Pods #

Finally, we can view the Pods. Shown in the figure below:

In the figure above, there are three Pods. We can see their names:

The first part, “nginx-deployment”, represents the name of the Pods’ Deployment. The middle part, “template-hash”, is the same for the three Pods because they are all created from the same template.

The last part of the name is a random string. By using the “get.pod” command, we can see that the ownerReferences of the Pod, which indicate the controller resource the Pod belongs to, are not Deployment but a ReplicaSet. The name of this ReplicaSet is “nginx-deployment” plus “pod.template-hash”. All Pods are created by a ReplicaSet, and the ReplicaSet corresponds to a specific version of the Deployment template.

Update Image #

Next, let’s see how to update the image version of all Pods in a given Deployment. We can execute the following command using “kubectl”:

kubectl set image deployment.v1.apps/nginx-deployment nginx=nginx:1.9.1

First, after “kubectl”, there is a fixed syntax “set image”, which indicates that we are setting the image. Secondly, “deployment.v1.apps” is also a fixed syntax, indicating the type of resource we want to operate on. “deployment” is the resource name, “v1” is the resource version, and “apps” is the resource group. It can also be abbreviated as “deployment” or “deployment.apps”. For example, when “deployment” is written, it defaults to using the “apps” group and the “v1” version.

The third part is the name of the deployment to be updated, which is our “nginx-deployment”. Next is the “nginx”, which refers to the template, or the container name in the Pod. Here, we can note that a Pod may have multiple containers, and the container.name that we specify to update the image refers to “nginx”.

Finally, we specify the image version that this container is expected to update to. Here, it is “nginx:1.9.1”. As shown in the figure below: after executing this command, we can see that the template.spec in the Deployment has been updated to “nginx:1.9.1”.

Quick Rollback #

If we encounter any problems during the release process, we can also perform a quick rollback. By running the “kubectl rollout undo” command, we can roll back to the previous version of the Deployment. By adding “rollout undo” and “to-revision”, we can specify a specific version to roll back to.

DeploymeStatus #

Finally, let’s take a look at DeploymeStatus. In previous lessons, we learned that each resource has its own spec.Status. Here, we can see that the three descriptions in deploymentStatus refer to its conversion status, which are “Processing”, “Complete”, and “Failed”.

Taking “Processing” as an example: “Processing” means that the Deployment is currently scaling and publishing. For example, a Deployment in the “Processing” state has all its replicas and Pod replicas reached the latest version and are available, which means it can enter the “Complete” state. If any scaling occurs in the “Complete” state, it will revert to the “Processing” state to handle the tasks.

If any problem occurs during the process, such as a failed image pull or readiness probe check, it will enter the “Failed” state. If a readiness probe check fails in the “Complete” state, the Deployment will also enter the “Failed” state. Once in the “Failed” state, the Deployment will only return to the “Complete” state when all replicas become available and are the latest updated versions.

Operation Demonstration #

Deployment Creation and Status #

Let’s start the operation demonstration: connect to an Alibaba Cloud service cluster here. We can see that there are several available nodes in the current cluster.

First, create the corresponding deployment. We can see that the desired, current, up-to-date, and available states of the deployment have all reached the expected status.

Structure of Deployment #

Here, we see that the number of replicas in the spec is three, and the labels defined in selector and template labels are all app: nginx. The image in the spec is the expected nginx: 1.7.9; the available.replicas, readReplicas, and updatedReplicas in the status are all three.

Pod Status #

Let’s select another Pod to check its status:

We can see that the functionality of ownerReferences in the Pod is ReplicaSet; the image in pod.spec.container is 1.7.9. This Pod is already in the Running state, and its conditions.status is “true”, indicating that its service is already available.

Update and Upgrade #

Currently, there is only the latest version of replicaset, so let’s try to upgrade the deployment.

The “kubectl set image” operation command is followed by “deployment”, followed by deployment.name, and finally specify the container name and the image version we want to upgrade to.

Next, let’s see that the image in the template in the deployment has been updated to 1.9.1.

Now let’s get the pod status.

The three pods have been upgraded to the new version, and the pod-template-hash in the pod names has also been updated.

As we can see, the spec and pod quantities of the old version replicaset are both 0, and the pod quantity of the new version is three.

Let’s assume an update is done again. This time, when we use get.pod, we can see that there are actually two old versions of pods in the running state, and another old version is being deleted; and two new versions of pods, one has entered the running state, and the other is still being created.

At this time, the available pod quantity is the quantity of pods that are not in the deleted state, which is actually four, exceeding the number of replicas originally set in the deployment, which is three. The reason for this is that we have operations called maxavailable and maxsugar in the deployment, and these two configurations can limit some strategies during the release process. This issue will be discussed in the later architectural design.

Preserve Historical Versions with revisionHistoryLimit #

From the above figure, we can see that the current latest version of replicaset is three pods, and there are also two historical versions of replicaset. Will there be a situation where the old version of replicaset keeps accumulating with the continuous updates of the deployment? In fact, deployment provides a mechanism to avoid this problem: in the deployment spec, there is a revisionHistoryLimit, which has a default value of 10. It actually ensures the number of replicasesets that can be retained in history versions. Let’s try changing it to 1.

From the second figure above, we can see that two replicaset versions are preserved, which means that in addition to the current version of replicaset, only one old version of replicaset is retained.

Rollback #

Finally, let’s try to roll back. First, let’s take a look at the replicaset. At this time, we found that the number of old versions of replicaset increased from 0 to 2, and the number of new versions of replicaset decreased from 3 to 1, indicating that it has started to roll back the operation. Then, let’s observe that the number of old versions is already 3, which means that the rollback has been successful, and the number of new version pods has become 0.

Finally, let’s get the pod status again:

At this time, the three pod.template-hash have been updated to the hash of the old version, but in fact, these three pods are newly created pods, rather than the 3 pods we created in the previous version. In other words, when we rolled back, we actually created 3 pods of the old version, rather than bringing back the previous 3 pods.

Architecture Design #

Management Mode #

Let’s take a look at the architecture design. First, let’s briefly discuss the management mode: Deployment is only responsible for managing different versions of ReplicaSets, while each ReplicaSet manages the specific number of Pod replicas. Each ReplicaSet corresponds to a version of the Deployment template. As seen in the previous example, whenever the template is modified, a new ReplicaSet is created, and the Pods under this ReplicaSet are actually of the same version.

As shown in the above diagram: Deployment creates ReplicaSets, and each ReplicaSet creates Pods. Their OwnerRef corresponds to the resources of their controller.

Deployment Controller #

Let’s first understand the implementation principle of the controller.

First, all our controllers utilize the events from Informer to perform certain Handlers and Watch. In the case of the Deployment controller, it focuses on the events in Deployment and ReplicaSet and adds them to a queue upon receiving them. After the Deployment controller retrieves an event from the queue, its logic will check if it is paused. This “paused” state indicates whether the Deployment needs a new release. If Paused is set to true, it means that the Deployment will only maintain a certain number and will not perform a new release.

As shown above, if Check paused is set to “Yes” or true, only Sync replicas will be performed. This means that the replicas will be synchronized to the corresponding ReplicaSet, and finally the Deployment status will be updated, completing the current ReplicaSet.

However, if the paused state is set to false, a Rollout will be performed. This means that the update is implemented through Create or Rolling. The way of updating is also implemented through Create/Update/Delete operations on ReplicaSets.

ReplicaSet Controller #

After the Deployment has assigned a ReplicaSet, the ReplicaSet controller also watches events from Informer, which include events from ReplicaSet and Pod. Once an event is retrieved from the queue, the ReplicaSet controller simply manages the number of replicas. If the controller finds that the number of replicas is greater than the number of Pods, it will scale up. On the other hand, if it finds that the actual number surpasses the expected number, it will delete Pods.

From the diagram of the Deployment controller shown above, it can be seen that the Deployment controller actually performs more complex tasks, including version management. It delegates the task of maintaining the number of each version to the ReplicaSet controller.

Scaling Simulation #

Let’s now simulate some operations, such as scaling. Here, we have a Deployment with 2 replicas, and there are two Pods, Pod1 and Pod2, under the corresponding ReplicaSet. If we modify the Deployment replicas, the controller will synchronize the replicas to the current version of the ReplicaSet. The ReplicaSet will create a new Pod, Pod3, because it detects that the current 2 Pods do not meet the desired 3 replicas.

Release Simulation #

Let’s simulate a release, which is slightly more complex. Here, we can see the initial template of the Deployment, for example, template1 version. Under this version of the ReplicaSet, there are three Pods: Pod1, Pod2, and Pod3.

When we modify the image of a container in the template, the Deployment controller will create a new ReplicaSet corresponding to template2. After creation, the ReplicaSet will gradually modify the number of Pods in both ReplicaSet1 and ReplicaSet2. For example, it will gradually increase the desired number of replicas in ReplicaSet2 and decrease the number of Pods in ReplicaSet1.

The end result is: the new version Pods are Pod4, Pod5, and Pod6, while the old version Pods have been deleted, completing the release.

Rollback Simulation #

Let’s take a look at the rollback simulation. Based on the previous release simulation, we know that Pod4, Pod5, and Pod6 have been successfully deployed. At this point, it is discovered that the current business version has a problem. Whether using the rollout command or rolling back by modifying the template, it actually rolls back the template to the old version, template1.

At this time, the Deployment will modify the desired number of Pods in ReplicaSet1, changing it to 3, and gradually decrease the number of replicas in the new version, ReplicaSet2. The final result is the re-creation of Pods from the old version.

Looking at the release simulation diagram, initially, Pod1, Pod2, and Pod3 are the old versions, and after the rollback, they become Pod7, Pod8, and Pod9. This means that the rollback does not retrieve the previous Pods, but rather creates new Pods that match the old version template.

Analysis of the spec field #

Finally, let’s briefly look at some fields in the Deployment spec. First, let’s look at the other spec fields in the Deployment:

• MinReadySeconds: Deployment checks the readiness of Pods to determine if they are available. However, if we set a value for MinReadySeconds, for example, 30 seconds, the Deployment will wait until Pods have been ready for more than 30 seconds before considering them available. The condition for a Pod to be available is that it must be ready, but a ready Pod may not necessarily be available. It must exceed MinReadySeconds to be considered available.
• revisionHistoryLimit: This field preserves the history of revisions and specifies the number of old ReplicaSets to keep. The default value is 10. It can be set to a number greater than 10 if there is a greater possibility of rollback.
• paused: The paused field is a flag that allows Deployment to maintain the desired number of replicas without making new releases. This can be useful in debug scenarios.
• progressDeadlineSeconds: As mentioned earlier, when the Deployment is in scaling or release states, its condition is in a processing state. Processing can have a timeout. If the processing exceeds the timeout, the controller will consider the Pod to have failed.

Analysis of the upgrade strategy field #

Finally, let’s take a look at the analysis of the upgrade strategy field.

The Deployment provides two strategies in the RollingUpdate section: MaxUnavailable and MaxSurge. The meanings of these two fields can be found in the detailed comments in the diagram or can be briefly explained as follows:

• MaxUnavailable: The maximum number of Pods that can be unavailable during the rolling process.
• MaxSurge: The maximum number of Pods that can exceed the expected replica count during the rolling process.

As mentioned earlier, for a Deployment with a ReplicaSet of 3, there may be a case during the release where both the new version ReplicaSet and the old version ReplicaSet each have two replicas, resulting in a total of 4 replicas, exceeding the desired count of 3. This is because the default values for MaxUnavailable and MaxSurge are both 25%. By default, during the release process, the Deployment can have 25% of replicas unavailable and can have up to 125% replicas, exceeding the desired count.

Users can adjust these values based on their specific scenarios. For example, if resources are sufficient and availability during the release process is a priority, MaxUnavailable can be set to a smaller value, while MaxSurge can be set to a larger value. However, if resources are limited, MaxSurge can be set to a smaller value or even 0. Note that MaxSurge and MaxUnavailable cannot both be set to 0.

The reason is not difficult to understand. When MaxSurge is set to 0, Pods must be deleted in order to scale up. If Pods are not deleted, new Pods cannot be scaled as this would result in a total number of Pods exceeding the desired count. If both MaxSurge and MaxUnavailable are set to 0, while MaxSurge ensures that no new Pods are scaled, MaxUnavailable cannot guarantee that there are available Pods in the ReplicaSet, which would cause problems. Therefore, both values cannot be set to 0. Users can set appropriate values based on their actual scenarios.

Summary #

The main content of this section is now concluded. Here is a brief summary:

• Deployment is a common workload in Kubernetes that supports deploying and managing multiple versions of Pods.
• The way Deployment manages multiple versions is by creating a ReplicaSet for each version’s template. The ReplicaSet maintains a certain number of Pod replicas, while the Deployment only needs to specify how many Pods in each version’s ReplicaSet.
• Therefore, the fundamental principle behind Deployment’s release and deployment is adjusting the desired number of replicas in different version ReplicaSets, in order to achieve the upgrade and rollback of multi-version Pods.