11 Observability Is Your Application Healthy Moriyuan #

The sharing of this course mainly revolves around the following five parts:

• Introduction to the sources of overall requirements;
• Introduction to the usage of Liveness and Readiness in K8s;
• Introduction to the diagnosis of common problems in K8s;
• Methods of remote debugging for applications;
• Course summary and practice.

Source of Requirements #

First, let’s take a look at the source of the entire requirement: how to ensure the health and stability of the application after migrating it to Kubernetes? In fact, it can be enhanced from two aspects:

1. Firstly, improve the observability of the application;
2. Secondly, improve the recoverability of the application.

In terms of observability, enhancements can be made in three areas:

1. Firstly, real-time observation of the health status of the application;
2. Secondly, obtaining information on the application’s resource usage;
3. Thirdly, obtaining real-time logs of the application for problem diagnosis and analysis.

When a problem occurs, the first thing to do is to reduce the scope of the impact and perform debugging and diagnosis. In the end, the ideal situation when encountering a problem is: complete recovery can be achieved through the self-healing mechanism integrated with Kubernetes.

Liveness and Readiness #

This section introduces Liveness probes and Readiness probes.

Application Health state - Introduction to Liveness and Readiness #

Liveness probe, also known as readiness probe, is used to determine whether a pod is in a ready state. When a pod is in a ready state, it can provide services externally, which means that traffic from the access layer can reach the corresponding pod. When this pod is not in a ready state, the access layer will remove the corresponding traffic from this pod.

Let’s take a look at a simple example:

The following figure is actually an example of a Readiness probe:

When the probe for this pod continually fails, the traffic from the access layer will not be directed to this pod.

When the state of this pod changes from FAIL to success, it can actually handle this traffic. Liveness probe is similar, it is the survival probe used to determine whether a pod is in a live state. What happens when a pod is not in a live state?

At this time, the upper-level judgment mechanism will determine whether this pod needs to be restarted. If the upper-level restart strategy is set to “restart always”, then this pod will be directly restarted.

Application Health state - Usage #

Next, let’s take a look at the specific usage of Liveness probes and Readiness probes.

Probe Types #

Liveness probes and Readiness probes support three different types of probes:

1. The first type is httpGet. It judges by sending an HTTP GET request. When the response code is between 200 and 399, it indicates that the application is healthy;
2. The second type is Exec. It judges whether the current service is normal by executing a command in the container. When the command returns 0, it indicates that the container is healthy;
3. The third type is tcpSocket. It performs a TCP health check by probing the IP and Port of the container. If the TCP connection can be established normally, it indicates that the container is healthy.

Probe Results #

From the probe results, there are mainly three types:

• The first type is success. When the status is success, it means that the container has passed the health check, that is, the Liveness probe or Readiness probe is in a normal state;
• The second type is Failure. Failure means that the container has not passed the health check. If the health check fails, appropriate actions will be taken. In the case of Readiness, the service layer will remove the pod that does not pass the Readiness check, while in the case of Liveness, the pod will be restarted or deleted.
• The third type is Unknown. Unknown means that the execution mechanism has not been fully executed, possibly because of timeouts or delays in returning scripts. At this time, Readiness-probe or Liveness-probe will not perform any operations and will wait for the next mechanism to verify.

In the kubelet, there is a component called ProbeManager. This component contains Liveness-probe or Readiness-probe, and these two probes will apply Liveness diagnosis and Readiness diagnosis to the pod to make a specific determination.

Application Health state - Pod Probe Spec #

Next, we will introduce the use of a yaml file for the different detection methods of these three types. First, let’s take a look at exec. The usage of exec is actually very simple. As shown in the figure below, you can see that this is a Liveness probe, which configures a diagnostic using exec. Next, it configures a field called command, which checks the status of the Liveness probe by cat-ing a specific file. When the result returned from this file is 0, or when the command returns 0, it will be considered that the pod is in a healthy state at that time.

Let’s take a look at httpGet. Inside httpGet, there is a field for the path, the second field is the port, and the third is headers. Sometimes, when you need to use a mechanism similar to header to check the health, you need to configure this header. In most cases, it may be sufficient to use health and port.

The third one is tcpSocket. The usage of tcpSocket is also quite simple, you just need to set the port to be monitored. In this example, it uses port 8080. When the tcp connection to this port is successfully established, the tcpSocket probe will consider the container to be in a healthy state.

In addition, there are five global parameters as follows:

• The first parameter is initialDelaySeconds, which represents the delay time for the first check after the pod is started. For example, for a Java application, it may take a longer time to start due to JVM startup and the loading of Java jar files. So initially, there may be a period of time when it cannot be detected, and this time is predictable, so you may need to set the initialDelaySeconds accordingly.
• The second parameter is periodSeconds, which represents the interval between checks. The default value is 10 seconds.
• The third parameter is timeoutSeconds, which represents the timeout for the checks. If the check is not successful within the timeout period, it will be considered a failure.
• The fourth parameter is successThreshold, which represents the number of successful checks required for a Pod to transition from failure to success. By default, it is set to 1, which means that if a probe originally failed, and the next probe is successful, the Pod is considered to be in a normal probe state.
• The last parameter is failureThreshold, which represents the number of consecutive probe failures before the Pod is considered to be in a failure state. The default value is 3, which means that if a Pod fails the probe 3 times consecutively while it was in a healthy state, it will be considered a failure.

Summary of Liveness and Readiness Probes #

Next, let’s summarize the Liveness probe and Readiness probe.

Introduction #

The Liveness probe is used to determine if a container is alive and whether the pod is running. If the Liveness probe determines that the container is unhealthy, the kubelet will kill the corresponding pod and determine whether to restart the container based on the restart policy. If the Liveness probe is not configured, it is considered to successfully pass the probe by default.

The Readiness probe is used to determine if the container has started and whether the pod’s condition is ready. If the probe result is unsuccessful, the pod will be removed from the endpoints, which means that traffic will be diverted away from the pod. It will only be added back to the corresponding endpoint when the next probe succeeds.

Failure Handling #

In the case of probe failures, the Liveness probe directly kills the pod, while the Readiness probe cuts off the association between the endpoint and the pod, meaning that traffic is diverted away from the pod.

Use Cases #

The Liveness probe is suitable for applications that can be restarted, while the Readiness probe is mainly for applications that cannot immediately provide services after startup.

Considerations #

When using Liveness and Readiness probes, there are some considerations to keep in mind. Because both Liveness and Readiness probes require appropriate probe configurations to avoid false positives.

• The first consideration is to increase the timeout threshold. This is because when executing a shell script inside a container, its execution time is usually longer. For example, a script that returns a result in 3 seconds on a regular ECS or VM may take 30 seconds inside a container. So this time needs to be evaluated within the container in advance. If possible, increasing the timeout threshold can prevent occasional timeouts due to high container load.
• The second consideration is to adjust the number of probes. The default value of 3 for successThreshold may not be optimal in short probe intervals. It is recommended to adjust the number of probe checks appropriately.
• The third consideration is that if exec is used for probing, the execution time may be long for shell scripts. It is recommended to use compiled scripts such as Golang or binaries compiled from C or C++ as they generally have 30% to 50% higher execution efficiency than shell scripts.
• The fourth consideration is that when using tcpSocket for probing, if you encounter TLS services, it may result in many incomplete TCP connections inside TLS. In this case, you need to evaluate whether this connection scenario will affect your business.

Problem Diagnosis #

Next, I will explain the common problem diagnosis in K8s.

Application Troubleshooting - Understanding State Mechanism #

First, it is important to understand the state mechanism in K8s. K8s is designed with a state machine concept. It uses YAML to define the desired state, and during execution, various controllers are responsible for transitioning between different states.

For example, in the above diagram, we can see the lifecycle of a Pod. Initially, it is in a pending state, and it can transition to running, unknown, or even failed. After running for some time, it can transition to succeeded or failed. In the case of an unknown state, it can be restored to running, succeeded, or failed due to state recovery.

Overall, the state transitions in K8s are based on a state machine mechanism, and the transitions between different states are represented by certain fields in the corresponding K8s objects, such as Status or Conditions.

The following diagram represents the state of a Pod.

For example, in a Pod, there is a field called Status, which represents the aggregated state of the Pod. In this example, the aggregated state is pending.

Moving further down, as a Pod can have multiple containers, each container has a State field representing the aggregated state of the container. In this example, the aggregated state is waiting because the image has not been pulled yet. The ready part is false because the container has not been successfully pulled, resulting in the Pod being unable to serve external requests. If the higher-level endpoint finds that the ready status is not true, the service cannot be accessed externally.

Next is the condition field. This mechanism represents smaller states within K8s that aggregate into the higher-level Status. In this example, there are several conditions. The first one is Initialized, which indicates whether the Pod has finished initialization. In this example, initialization is complete, and it moves on to the ready state. Since the containers haven’t pulled the corresponding image, the ready status is false.

Moving further down, we can see whether the container is ready. In this case, it is false, and the status is PodScheduled, indicating whether the Pod is scheduled to a node and bound to it. In this case, it is true.

By checking if the corresponding conditions are true or false, we can determine the overall status. Different state transitions in K8s generate events, which can be either normal or warning events. In the first event shown, it is a normal event, and the reason is scheduler, indicating that the Pod has been scheduled to the default scheduler and placed on the node cn-beijing192.168.3.167. Next, there is a normal event, indicating that the current image is being pulled. Then, there is a warning event, indicating that the image pull has failed.

Similarly, in Kubernetes, the transition between these states generates corresponding events, which are exposed in a similar way as “normal” or “warning”. Developers can use these events to determine the specific state of the application and perform a series of diagnostics using the upper-level condition status and related fields.

Troubleshooting Common Application Abnormalities #

This section introduces some common abnormalities that may occur in applications. First, let’s talk about the states that a pod can be in.

Pod Stuck in Pending #

The first state is “pending”, which means that the scheduler has not intervened. You can use kubectl describe pod to view the corresponding events. If the pod cannot be scheduled due to resource or port occupation, or due to node selector constraints, the results will be displayed in the events. The results will show how many nodes do not meet the requirements, how many do not meet the CPU requirements, how many do not meet the node requirements, and how many do not meet the requirements due to tag labels.

Pod Stuck in Waiting #

The second state is when the pod is stuck in the “waiting” state. When the pod’s states are in waiting, this usually indicates that the pod’s image was not pulled successfully. The reasons for this can be: the image is a private image but no pod secret is configured, the image address does not exist and cannot be pulled, or the image is a public image and the pulling process fails.

Pod Continuously Restarting and Crashing #

The third situation is when a pod keeps restarting and you can see something like “backoff”. This usually means that the pod has been scheduled but failed to start. In this case, you should pay attention to the specific status of the application itself, not whether the configuration or permissions are correct. At this time, you should check the specific pod logs.

Pod Running but not Working Properly #

The fourth situation is when a pod is in the “running” state but is not serving properly. In this case, a common issue may be due to fine-grained configuration errors, such as misspelling certain fields. This may cause the YAML to be applied but not take effect correctly, resulting in the pod being in the “running” state but not serving properly. You can use the kubectl apply-validate-f pod.yaml command to check if the YAML is correct. If there are no issues with the YAML, the next step is to diagnose whether the port configuration and Liveness or Readiness are correctly set.

Service Not Working Properly #

The last situation is when a service is not working properly. How to determine this? When service issues occur, it is often due to issues with how it is being used. The relationship between a service and the underlying pod is established through selectors. This means that the pod is configured with certain labels, and the service matches these labels to associate with the pod. If there is an issue with the label configuration, it may cause the service to fail to find the corresponding endpoints, resulting in the service not being able to provide external access. If a service has issues, the first thing to check is whether there is a valid endpoint behind the service, and then check if this endpoint can provide normal services externally.

Remote Debugging of Applications #

This section explains how to perform remote debugging of applications in K8s, including remote debugging of pods, service, and performance optimization.

Remote Debugging of Applications - Remote Debugging of Pods #

When deploying an application in a cluster and encountering problems, it is essential to perform quick verification or make modifications. In such cases, it is often necessary to log in to the container for diagnosis.

For example, you can enter a pod using the exec command. In this command, you can use kubectl exec -it pod-name followed by an appropriate command, such as /bin/bash, to enter an interactive bash shell within the pod. Then, you can execute various commands in the shell, such as modifying configurations or restarting the application through supervisor.

What if there are multiple containers within a pod? How can we specify the container within the pod? In this case, you can use the -c parameter as shown in the command at the bottom of the figure. After the -c parameter, specify the container-name within the pod to enter and then add the specific command to achieve remote debugging of multiple containers.

Remote Debugging of Applications - Remote Debugging of Services #

How do we perform remote debugging of services? Remote debugging of services can be further divided into two scenarios:

• The first scenario involves exposing a service to a remote cluster, allowing applications in the remote cluster to call the local service. This is a reverse link.
• The second scenario involves allowing the local service to call a remote service. This is a forward link.

For the reverse link scenario, there is an open-source component called Telepresence that enables proxying of the local application to a service in a remote cluster. Using Telepresence is straightforward.

First, deploy the Telepresence Proxy application to the remote K8s cluster. Then, swap the remote deployment with the local application using the command telepresence swap deployment DEPLOYMENT_NAME. This way, you can proxy a local application onto a remote service and debug the application within the remote cluster. For those interested, you can find more information about the usage of this plugin on GitHub. The second method is to use port-forwarding when a local application needs to call a service in a remote cluster. For example, if there is an API server in the remote cluster that provides some ports, and you want to directly call this API server when debugging code locally, a simple way to do this is through port-forwarding.

The way to use it is kubectl port-forward, followed by the remote service name of the service, the corresponding namespace, and additional parameters such as port mapping. Through this mechanism, a remote application can be proxied to a local port, and the remote service can be accessed by accessing the local port.

Open-source debugging tool - kubectl-debug #

Finally, let me introduce an open-source debugging tool, called kubectl-debug. We know that in Kubernetes, the commonly used container runtimes at the underlying level are Docker and containerd. Both Docker and containerd use Linux namespaces for virtualization and isolation.

Usually, there won’t be many debugging tools included in the images, such as netstat and telnet, because they would make the application very bloated. So how can we debug in such cases? This is where tools like kubectl-debug come in handy.

kubectl-debug relies on Linux namespaces to detach a Linux namespace from a container and execute any debugging actions within this container. It is essentially the same as directly debugging the Linux namespace. Here is a simple operation to explain:

This kubectl-debug plugin has already been installed. So, you can directly diagnose a remote pod using the kubectl-debug command. In this example, when you execute the debug command, it will first pull some images. These images actually come with some default diagnostic tools. When this image is enabled, the debug container will be started. At the same time, this container will attach to the corresponding namespace of the container you want to diagnose. This means that this container and the container you want to diagnose are in the same namespace, so you can view real-time information related to the network or kernel parameters in this debug container.

In this example, you can see information related to the hostname, processes, netstat, and so on. These are all in the same environment as the pod being debugged, so you can obtain relevant information with these three commands.

If you log out at this time, it will kill the corresponding debug pod and exit. This will not have any impact on the application. So, with this method, you can achieve diagnosis without interfering with the containers.

In addition, it also supports some additional mechanisms. For example, you can set an image and install tools such as htop. Developers can define their own command line tools through this mechanism and set them up using images. In this way, you can debug a remote pod using this mechanism.

Summary of this section #

• About Liveness and Readiness probes. The Liveness probe is used to check if a pod is alive, while the Readiness probe is used to determine if a pod is ready. If a pod is ready, it can provide services externally. This is the important part to remember about Liveness and Readiness.
• Three steps for application diagnosis: First, describe the corresponding status. Then, use the status to troubleshoot specific diagnostic directions. Finally, check the events of the corresponding object to obtain more detailed information.
• Provide logs for pods to locate the application’s own status.
• Remote debugging strategy: If you want to proxy a local application to a remote cluster, you can use tools like Telepresence. If you want to proxy a remote application to the local environment for calling or debugging, you can use mechanisms like port-forward.