28 Application Protection How to Keep Pods Running Healthily

28 Application Protection How to Keep Pods Running Healthily #

Hello, I’m Chrono.

In the previous lessons, we mostly focused on how to use various API objects to manage and manipulate Pods, with less attention on the Pods themselves.

As a core concept and atomic scheduling unit in Kubernetes, Pods are responsible for managing containers and represent applications in the form of logical hosts, container collections, and process groups. Their importance is self-evident.

Now, let’s take a step back and look at two methods of configuring Pods in Kubernetes on top of the aforementioned higher-level API objects: resource quotas and health checks. These methods can provide various assurance measures for Pods, ensuring that applications run healthier.

Container Resource Quotas #

As early as [Lesson 2], we mentioned that there are three major isolation technologies for creating containers: namespace, cgroup, and chroot. Among them, namespace implements independent process space, chroot implements independent file system, but we have not seen the specific application of cgroup.

The role of cgroup is to control CPU and memory, ensuring that containers do not indiscriminately occupy basic resources and thus affect other applications in the system.

However, containers always need to use CPU and memory. How to handle the relationship between requirements and limitations of these two?

Kubernetes’ approach is somewhat similar to the usage of PersistentVolumeClaim we mentioned in [Lesson 24], which is that the container needs to first make a “written application”, and Kubernetes decides whether to allocate resources and how to allocate them based on this application.

However, there are obvious differences between CPU, memory, and storage volumes, because CPU and memory are directly “built-in” in the system, unlike hard disks that need to be “attached”, so the process of application and management will be much simpler.

The specific application method is very simple, just add a new field resources in the description section of the Pod container, which is equivalent to a resource Claim application.

Let’s take a look at a YAML example:

apiVersion: v1
kind: Pod
metadata:
  name: ngx-pod-resources

spec:
  containers:
  - image: nginx:alpine
    name: ngx

    resources:
      requests:
        cpu: 10m
        memory: 100Mi
      limits:
        cpu: 20m
        memory: 200Mi

This YAML file defines an Nginx Pod. What we need to focus on is containers.resources, which has two fields:

  • “requests”, which means the resources the container needs to request, that is to say, Kubernetes must allocate the resources listed here when creating the Pod, otherwise the container cannot run.
  • “limits”, which means the upper limit of resources used by the container, and it cannot exceed the set value, otherwise it may be forcibly stopped.

When requesting CPU and memory resources, you need to pay special attention to their representation.

The memory notation is the same as disk capacity, using Ki, Mi, Gi to represent KB, MB, GB, for example, 512Ki, 100Mi, 0.5Gi, etc.

As for CPU, because the quantity is limited in computers and is very precious, Kubernetes allows containers to finely divide CPU. It can fully use 1 or 2 CPUs, or use fractions such as 0.1 and 0.2 to partially use CPU. This is actually similar to the usage of UNIX “time slice”, which means how much CPU time the process can occupy at most.

However, CPU time cannot be divided infinitely. The minimum unit of CPU usage in Kubernetes is 0.001, and a special unit m is used for easy representation, which means “milli”. For example, 500m is equivalent to 0.5.

Now let’s take a look at this YAML again. You should understand that it is applying for 1% of CPU time and 100MB of memory from the system, and the resource limit during runtime is 2% of CPU time and 200MB of memory. With this application, Kubernetes will find the node that best meets the resource requirements in the cluster to run the Pod.

Here is an animated picture I found online. Based on the demands declared by each Pod, Kubernetes will “fit” the nodes as much as possible, just like building with building blocks or playing Tetris, fully utilizing the resources of each node and maximizing the efficiency of the cluster.

Image

You may have a question: What if the Pod does not write the resources field? How will Kubernetes handle it?

This means that the Pod has “no lower limit or upper limit” on the resource requirements for running, and Kubernetes does not need to consider whether the CPU and memory are sufficient. It can schedule the Pod to any node, and the Pod can also use CPU and memory without any restrictions during runtime.

In our course environment, there is no problem with this approach, but it would be dangerous in a production environment. The Pod may run slowly due to insufficient resources or occupy too many resources and affect other applications. Therefore, we should evaluate the resource usage of the Pod reasonably and try to add restrictions to the Pod.

At this point, you may continue to ask: What will happen if the estimated resource demand of the Pod is too high and the system cannot meet it?

Let’s try it. First, delete the resource limit resources.limits of the Pod and change resources.request.cpu to an extreme value of “10”, which means it requires 10 CPUs:

...

resources:
  requests:
    cpu: 10

Then use kubectl apply to create this Pod. You may be surprised to find that although our Kubernetes cluster only has 3 CPUs, the Pod is still created successfully.

However, if we use kubectl get pod to check, we will find that it is in the “Pending” state, and it is actually not scheduled and running:

Image

Use the command kubectl describe to view the specific reason, and you will find such a prompt:

Image

This explicitly tells us that Kubernetes scheduling has failed, and all nodes in the current cluster cannot run this Pod because it requires too many CPUs.

What is a Container State Probe #

Now, with the use of the resources field and resource quotas, the running of Pods in Kubernetes has some basic guarantees. Kubernetes monitors the resource usage of the Pod, ensuring that it neither starves nor becomes overwhelmed.

However, this is only the most basic level of assurance. If you have developed or operated actual backend services, you will know that even if a program starts successfully, it may still fail to provide services for various reasons. The most common scenario is a failure such as a “deadlock” or “infinite loop” at runtime. From the outside, the process appears normal, but internally it is a mess.

Therefore, we hope that Kubernetes, as a “caretaker”, can monitor the state of the Pod more carefully. In addition to crash recovery, it must be able to probe the internal running state of the Pod, perform regular “check-ups” for the application, and keep the application constantly “healthy” and able to work stably under heavy load.

So, how should we check the health status of the application?

Because applications come in various forms, they are like a black box to the outside world. We can only see the three basic states of startup, running, and shutdown. Apart from this, there is no good way to know if they are working properly internally.

Therefore, we need to turn applications into gray boxes, making some internal information visible to the outside. This way, Kubernetes can probe the internal state.

Speaking of which, the probing process is a bit like the nucleic acid testing we are familiar with now. Kubernetes uses a cotton swab to extract data from the “testing port” of the application to judge its “health” based on this information. This feature is aptly named “probe”, or “detector”.

Kubernetes defines three types of probes to check the state of the application, each corresponding to a different state of the container:

  • Startup Probe, used to check if the application has started successfully. It is suitable for applications with a lot of initialization work and a slow startup process.
  • Liveness Probe, used to check if the application is running properly and whether there are any deadlocks or infinite loops.
  • Readiness Probe, used to check if the application is ready to receive traffic and provide services.

You need to note that these three probes have a progressive relationship: the application starts with the Startup state, after completing the basic initialization work such as loading configuration files, it enters the Liveness state if there are no exceptions. However, there may still be some preparation work that has not been completed, so it may not be able to provide services externally. Only when it reaches the Readiness state is the container in its healthiest and most available state.

Understanding these three states may be a bit difficult at first. I have drawn a diagram showing the corresponding relationship between the states and the probes. You can take a look:

Image

So how does Kubernetes use the states and probes to manage containers?

If a container in a Pod is configured with probes, Kubernetes will continuously call the probes to check the container’s state after starting:

  • If the Startup Probe fails, Kubernetes will consider the container to have not started properly and will attempt to restart it repeatedly. Of course, the Liveness and Readiness Probes will not be started either.
  • If the Liveness Probe fails, Kubernetes will consider the container to be in an abnormal state and will restart it.
  • If the Readiness Probe fails, Kubernetes will consider the container to be running but with internal errors, unable to provide services properly. Kubernetes will exclude the container from the load balancing pool of the Service object, and no traffic will be allocated to it.

Now that we know how Kubernetes handles these three states, we can write appropriate checking mechanisms when developing applications, allowing Kubernetes to perform regular “check-ups” for the application using probes.

Based on the previous diagram, I have added the actions taken by Kubernetes. By looking at this diagram, you can better understand the workflow of container probes:

Image

How to use container probes #

Now that we understand resource quotas and the concept of probes, let’s dive into the main part of today’s discussion and learn how to define probes in a Pod’s YAML description file.

The configuration for startupProbe, livenessProbe, and readinessProbe is identical, with a few key fields:

  • periodSeconds: The interval at which the probe should be executed. The default is 10 seconds.
  • timeoutSeconds: The timeout period for the probe. If the probe exceeds this timeout, it is considered a failure. The default is 1 second.
  • successThreshold: The number of consecutive successful probe results required to consider the probe successful. For startupProbe and livenessProbe, this value can only be 1.
  • failureThreshold: The number of consecutive probe failures required to consider the probe a failure. The default is 3.

As for the probe types, Kubernetes supports three: Shell, TCP Socket, and HTTP GET. They can be configured in the probe as follows:

  • exec: Executes a Linux command, such as ps or cat. This option is similar to the command field for containers.
  • tcpSocket: Attempts to establish a TCP connection to a specific port in the container.
  • httpGet: Connects to a port and sends an HTTP GET request.

To use these probes, we need to reserve a “check endpoint” when developing the application. This allows Kubernetes to make use of the probes and gather information. Let’s take Nginx as an example and create a ConfigMap using a configuration file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ngx-conf
data:
  default.conf: |
    server {
      listen 80;
      location = /ready {
        return 200 'I am ready';
      }
    }

If you’re not familiar with Nginx configuration syntax, let me briefly explain. In this configuration file, we enable port 80 and use the location directive to define an HTTP path called /ready. This path serves as the “check endpoint” exposed to the outside world to verify the readiness status. It returns a simple 200 status code and a string indicating that everything is working fine.

Now, let’s take a look at the specific probe definitions inside the Pod:

apiVersion: v1
kind: Pod
metadata:
  name: ngx-pod-probe
spec:
  volumes:
  - name: ngx-conf-vol
    configMap:
      name: ngx-conf
  containers:
  - image: nginx:alpine
    name: ngx
    ports:
    - containerPort: 80
    volumeMounts:
    - mountPath: /etc/nginx/conf.d
      name: ngx-conf-vol
    startupProbe:
      periodSeconds: 1
      exec:
        command: ["cat", "/var/run/nginx.pid"]
    livenessProbe:
      periodSeconds: 10
      tcpSocket:
        port: 80
    readinessProbe:
      periodSeconds: 5
      httpGet:
        path: /ready
        port: 80

The startupProbe uses the Shell mode and checks the existence of the Nginx process ID file (/var/run/nginx.pid) on disk using the cat command. If the file exists, the probe considers the startup successful. It is executed every second.

The livenessProbe uses the TCP Socket mode and attempts to connect to port 80 of the Nginx container. It is executed every 10 seconds.

The readinessProbe uses the HTTP GET mode and accesses the /ready path within the container. It sends a request every 5 seconds.

Now, let’s create this Pod using kubectl apply and examine its status:

Image

Since this Nginx application is very simple, the probe checks will always be successful once it starts. You can use kubectl logs to view the Nginx access logs, which will contain the execution status of the HTTP GET probe:

Image

From the screenshot, you can see that Kubernetes sends an HTTP request to the /ready URI approximately every 5 seconds, continuously checking the container’s readiness state.

To verify the functionality of the other two probes, you can modify the probes to check incorrect files or ports, for example:

    startupProbe:
      exec:
        command: ["cat", "nginx.pid"]  # Incorrect file
    livenessProbe:
      tcpSocket:
        port: 8080  # Incorrect port

Then recreate the Pod and observe its status.

When the startupProbe fails, Kubernetes will continuously restart the container. You will notice that the RESTARTS count keeps increasing, and the livenessProbe and readinessProbe will not be executed. Although the Pod remains in the Running state, it will never become READY:

Image

Since the default failureThreshold count is three, Kubernetes will perform three consecutive livenessProbe TCP Socket checks at intervals of 10 seconds. Only when all three checks fail after 30 seconds, the container will be restarted:

Image

Feel free to modify the readinessProbe and see how it affects the Pod’s state.

Summary #

Alright, today we learned about two ways to configure the runtime assurance of Pods: Resources and Probes. Resources involve setting resource limits for containers, while Probes involve active health checks that allow Kubernetes to monitor the running state of applications in real time.

To recap today’s content:

  1. Resource quotas use cgroup technology to limit the CPU and memory usage of containers, enabling Pods to make optimal use of system resources and making it easier for Kubernetes to schedule Pods.
  2. Kubernetes defines three types of health probes: Startup, Liveness, and Readiness probes. These probes respectively detect the startup, liveness, and readiness status of applications.
  3. Probes can use Shell, TCP Socket, or HTTP Get methods for detection, and parameters such as frequency and timeout can be adjusted for the probes.

Homework #

Finally, it’s time for homework. I have two questions for you to consider:

  1. Can you explain the difference between Liveness and Readiness probes?
  2. What are the advantages and disadvantages of using Shell, TCP Socket, and HTTP GET as probing methods?

Feel free to leave your comments in the comment section below. The course is coming to an end, thank you for your perseverance in learning for so long. See you in the next class.

Image