38 Plus Lesson 8 Some Pitfalls When Java Programs Migrate From Vm to Kubernetes

38 Plus Lesson 8 Some Pitfalls When Java Programs Migrate from VM to Kubernetes #

Here we meet again. The end of the course does not mean the end of our journey. I am delighted to continue sharing valuable content with you, and I hope you will keep communicating with me in the comments section to share your learning experiences and practical knowledge.

By deploying applications on Kubernetes at a large scale, overall resource utilization can be improved, cluster stability can be enhanced, and rapid cluster scaling capabilities can be provided. It can even enable automatic cluster scaling based on workload pressure. Therefore, more and more companies are now migrating their programs from virtual machines (VMs) to Kubernetes.

In most companies, the Kubernetes cluster is set up by the operations team, and program deployment is usually handled by a CI/CD platform. The entire migration process from virtual machines to Kubernetes generally does not require any code modification, only a re-release may be needed. As a result, Java developers may not be very aware of the migration itself and consider Kubernetes as something only the operations team needs to know about. However, once a program is deployed in a Kubernetes cluster and runs in a container environment, various strange issues that have never been encountered before will inevitably arise.

In today’s extra meal, let’s take a look at some of the “pitfalls” you may encounter during this process and the corresponding “avoidance methods.”

Pitfall of Unfixed Pod IP #

Pod is the smallest unit that can be created and deployed in Kubernetes. We can access a specific application instance through the Pod IP. However, it is important to note that the Pod IP is not fixed and will change after the Pod restarts, unless specifically configured.

Fortunately, most of our Java microservices are stateless, so we don’t need to access a specific Java service instance through the Pod IP. Generally, there are two ways to discover and access microservices deployed in Kubernetes:

Implement it through Kubernetes. This means using a Service for internal service communication and using Ingress to access the service cluster from outside.
Implement it through a microservice registry (such as Eureka). This means that service-to-service communication is achieved through client-side load balancing, and direct access to the Pod IP is used. External access to the service cluster is routed through a microservice gateway.

When accessing microservices using these two methods, we don’t directly deal with Pod IPs, and we don’t persistently record Pod IPs. Therefore, we generally don’t need to pay too much attention to the issue of Pod IP changes. However, Pod IP changes can cause some problems in certain scenarios.

I have encountered a situation where a task scheduling middleware records the IP of the scheduled node in the database, and then tries to access the node’s execution logs by visiting the node’s IP. If the node is deployed in Kubernetes, the Pod IP will change after the node restarts. As a result, the previously recorded Pod IP of the old node cannot be accessed, leading to the inability to view task logs.

In such cases, what should we do? At this time, it may be necessary to modify the middleware to persist the task execution logs, thereby avoiding the need to access the task node to view the logs.

In conclusion, we need to be aware of the issue of unfixed Pod IP and take precautions. Before migrating to a Kubernetes cluster, it is important to assess whether there will be a need to access old nodes using their IP addresses. If there is, appropriate modifications should be made.

Pitfalls of Program Being Killed due to OOM #

When deploying programs in a Kubernetes cluster, we usually set a certain memory limit for containers. Containers cannot use more resources than what is set by their resource “limit” attribute. If a Java program inside the container uses a large amount of memory, various OOM (Out of Memory) situations may occur.

The first case is the OS OOM Kill issue. If excessive memory usage destabilizes the operating system kernel, the operating system may kill the Java process. In this case, you can find keywords like “oom_kill_process” in the operating system’s /var/log/messages log.

The second case is the most common OOM issue with Java programs. When the program exceeds the limited heap memory and requests more memory, a Heap OOM occurs. Subsequently, the Pod may be restarted by Kubernetes due to failed health checks.

When deploying a Java program in Kubernetes, both of these cases are common, and they are manifested by the keywords OOM + restart. Therefore, when the operations team mentions that the program was killed or restarted due to OOM, we must work together with them to differentiate which case it is, and then address it accordingly.

For case 1, the cause of the problem is often not a lack of Java heap memory, but rather the program using too much off-heap memory that exceeds the memory limit. In this case, increasing the maximum heap memory of the JVM will only make the problem worse because the heap memory can be reclaimed by garbage collection (GC). We need to analyze which part of the Java process is consuming excessive memory, whether it is reasonable, and whether there may be memory leaks. In addition to the heap, the memory consumption of a Java process also includes:

Direct memory

Metaspace

Thread stack size (Xss) * number of threads

JIT code cache

GC and compiler additional space

…

We can use NMT (Native Memory Tracking) to print the size of each memory region to determine which part of the memory is occupying too much memory or if there may be memory leaks:

java -XX:NativeMemoryTracking=summary/detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintNMTStatistics

If you confirm that it is case 2 for the OOM, I still do not recommend increasing the heap memory limit directly to prevent case 1 from happening again. Instead, I suggest setting the heap memory limit to 50% to 70% of the container memory limit, leaving enough memory for off-heap and OS kernel. If you need to increase the heap memory, you should also increase the container’s memory limit accordingly. Additionally, you should investigate why the heap memory usage is so high and eliminate the possibility of memory leaks by using techniques such as Heap Dump (you can review the related content in Lesson 35).

Pitfalls of Memory and CPU Resource Configuration Mismatch in Containers #

As we mentioned earlier, heap memory scaling needs to be synchronized with the memory limit of the container. However, what we would prefer is for the heap memory configuration of Java programs to automatically scale up or down based on the resource configuration of the container, rather than being hardcoded with Xmx and Xms. This way, operations personnel can easily scale up or down the entire cluster.

For JDK versions > 8u191, we can set the following JVM parameters to automatically adjust the heap memory allocation based on the container memory limit. For example, the following configuration sets both Xmx and Xms to 50% of the container memory limit:

-XX:MaxRAMPercentage=50.0 -XX:InitialRAMPercentage=50.0 -XX:MinRAMPercentage=50.0

Moving on, let’s take a look at the pitfalls of CPU resource configuration mismatch in containers and the corresponding solutions.

When it comes to CPU resource utilization, we mainly need to pay attention to the fact that various components in the code, including the JVM itself, configure important parameters based on the number of CPUs. Therefore:

If this value is obtained by the JVM from the number of Kubernetes worker nodes due to compatibility issues, then the number may not be 4 or 8, but may exceed 128, resulting in excessively high concurrency.

For JDK versions > 8u191, although compatibility with containers may be better, the value obtained from Runtime.getRuntime().availableProcessors() is actually the value of the request rather than the limit (for example, if we set the request as 2 and the limit as 8, CICompilerCount and ParallelGCThreads may only be 2). As a result, the concurrency may be too low, thereby affecting the GC or compilation performance of the JVM.

Therefore, my suggestions are:

First, use the -XX:+PrintFlagsFinal switch to confirm whether ActiveProcessorCount meets our expectations, and also to confirm whether important parameter configurations such as CICompilerCount and ParallelGCThreads are reasonable.

Second, set the CPU’s request and limit to be the same, or for JDK versions > 8u191, directly set ActiveProcessorCount to the CPU limit of the container using -XX:ActiveProcessorCount=xxx.

Restarting Pods and Pitfalls of Losing the Context After Restart #

Unless there are issues with the host machine, virtual machines are not likely to restart by themselves or be restarted. However, Pod restarts in Kubernetes are not uncommon. Pods will restart in cases where the liveness check fails or when the Pod is rescheduled to a different node. When it comes to Pod restarts, there are two issues we need to pay attention to.

The first issue is to analyze why a Pod is being restarted.

In addition to the OOM issue mentioned earlier, where the program is killed due to out-of-memory, we also need to consider cases where the liveness check fails.

Kubernetes has two probes: readinessProbe and livenessProbe. The readinessProbe checks whether the application has started, while the livenessProbe performs continuous health checks. Normally, these probes are configured as breakpoints for health checks. If a health check takes a long time to complete (e.g., due to storage or external service availability checks), it is possible for the readinessProbe to pass but the livenessProbe to fail (since we usually set a longer timeout for the readinessProbe compared to the livenessProbe). Moreover, the health check may be affected by Full GC, causing timeouts. Therefore, we need to work together with the operations team to confirm whether the configuration of the livenessProbe, including the address and timeout settings, is reasonable to prevent Pod restarts due to intermittent livenessProbe failures.

The second issue is to understand the difference between Pods and virtual machines.

Virtual machines are usually stateful, so even if a Java program deployed on a virtual machine restarts, we still have access to the previous context. However, in the case of Pod restarts, a new Pod is created, which means that the old Pod becomes inaccessible. Therefore, if a restart occurs due to heap OOM issues, it becomes unlikely to be able to view some system logs at that time or execute commands on the previous context.

Therefore, we need to find ways to preserve the context as much as possible before a Pod is terminated. For example:

Application logs, standard output, GC logs, and other relevant files can be directly mounted as persistent volumes instead of being stored within the container.
For preserving heap stack traces, we can configure -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath to generate a heap dump when an OOM occurs. We can also have the JVM invoke a shell script to preserve thread stack information:

-XX:OnOutOfMemoryError=saveinfo.sh

For preserving the container’s context, we can have the operations team configure a preStop hook to upload necessary information to a persistent volume or a cloud storage platform before the Pod is terminated.

Key Takeaways #

Today, we discussed four types of issues that may occur when deploying Java applications to a Kubernetes cluster.

The first type of issue is that we need to understand that the IP of the application will change dynamically. Therefore, we need to design our application without strong dependencies on Pod IP, and use service discovery to locate the application.

The second type of issue is that when encountering OOM (Out of Memory) problems, we need to distinguish whether the problem is caused by the Java process or the container itself. If it is a container-level issue, we need to further analyze which memory region is using excessive memory. After identifying the problem, we can adjust JVM parameters or scale up resources according to container resource settings.

The third type of issue is to ensure that the memory and CPU resources used by the application match the container’s resource limits. We need to ensure that the program’s view of host resources is based on the container itself rather than the physical machine. Additionally, the program should scale its resource limits as the container scales.

The fourth type of issue is that we need to pay close attention to program restarts outside the deployment period. We should prepare for Pod restarts by eliminating possibilities such as improper resource configuration or failed health checks. This helps to avoid occasional performance or availability issues caused by frequent program restarts.

Only by addressing these hidden risks can the Kubernetes cluster perform better.

Reflection and Discussion #

Have you encountered any other pitfalls in Java+Kubernetes in your work?

I am Zhu Ye. Feel free to leave a comment and share your thoughts. You are also welcome to share today’s content with your friends or colleagues for further discussion.