24 Practice Chapter- Kubernetes (K8s) Cluster Configuration Testing #
The second part of the content breaks down the installation and configuration of Kubernetes core components: Containerd, Kubeadm, IPVS, Calico, kube-dns. Readers can independently configure a cluster based on the official documentation. However, due to the complexity of cluster configuration, the resulting environments vary greatly, making it difficult to achieve a unified understanding. The purpose of this quiz is to verify the knowledge we have learned and build a cluster together, thereby verifying our understanding of cluster concepts.
Environment #
Since the first day we encountered container technology, our goal has been to solve the problem of environment dependencies. Docker wraps the environment together with the application into an image for distribution. When configuring a Kubernetes cluster, the smallest unit of operation is a Pod, which can be understood as a group of containers. But this group of containers is not simply a collection of containers. It cleverly designs a basic container with a pause as its core, which contains all the environment dependencies of the related applications within its runtime. Other related business containers are added to this runtime, and problems with these business containers will not affect the environment. This is the core design of Kubernetes for building business clusters, which ingeniously addresses the issue of application service availability.
Now we need to choose the version of the operating system. You will find that official documentation does not specify a particular version. In fact, there is no such convention. The purpose of containers is to resolve environment dependencies. However, after years of evolution, to be more precise, we still have a core dependency issue related to the kernel. The characteristics of the kernel determine the characteristics of containers. When choosing a version, we generally refer to the version of Docker, such as 18.09 or 19.03, which are the most popular.
As you may have noticed, you cannot guarantee that these Docker versions will be problem-free in specific environments. This is the pit we dig for ourselves when configuring production environments. If you are using it internally within an organization, the best approach is to establish a baseline, define version numbers, and invest resources to maintain the stability of this version based on extensive practice. Container technology is evolving rapidly, and now Kubernetes and Docker are increasingly avoiding each other. Both are using containerd to support the management of container runtimes at the underlying level. As users, we cannot avoid this.
This highlights another issue: with the evolution of components, which version should I choose, and are they stable? Since Kubernetes is driven by an open-source community, we must use these software in accordance with open-source practices to gain accurate experience. Based on my summary of experiences, I have some suggestions for your reference: x86-64 is still the best system architecture for containers. The mainstream systems are mainly RedHat/CentOS 7.x and Ubuntu 16.04. For the RedHat series, the kernel version should be 3.10 and above, while Ubuntu can go up to 4.4. Some users upgrade the kernel to 4.4 by using the open-source kernel repository, which is also quite common. However, upgrading the kernel introduces many unknown modules and makes the system unstable. The ARM system architecture also has compatibility requirements for the file format of the entire Kubernetes components. When choosing an appropriate architecture, it is important to ensure that the corresponding Kubernetes components are ready. In summary, the mainstream operating systems are mainly RedHat 7.x and Ubuntu LTS 16.04. Upgrading to a major version of the operating system requires a lot of adaptation work for Kubernetes, and this is something the open-source community is unlikely to do for users. Pay attention to this.
The versions of Kubernetes are updated quickly, and the community maintains three mainline versions. Currently, the main versions are 1.16.x, 1.17.x, and 1.18.x. Each of these “x” versions iterates approximately every two weeks and mainly focuses on bug fixes. Many teams have summarized some tricks, such as using odd or even versions as their main versions, in order to avoid the instability brought by the latest version. It is not to say that odd versions are good or even versions are stable, that is just pure speculation. As an open-source software, its quality is maintained by the community, and as users, we are all guinea pigs who need to test and verify the reliability of the components in our own environments. In summary, the mainstream environment still chooses to use a version that is one or two sub-versions lower than the latest version as the software to be maintained. Maintaining open-source software is not free, it requires the efforts of everyone to ensure the reliability of the components.
In addition to delaying the adoption of one to two versions for the mainline version of Kubernetes, for other auxiliary components such as Calico, kube-dns, Containerd, etc., it is recommended to use the latest version. The main reason is that they are frontline components that are called more frequently, which means there are more opportunities to discover issues. The earlier an issue is discovered, the faster it can be fixed. This is a principle in the open-source world: the earlier an issue is discovered, the earlier it can be fixed, and the more stable the component becomes. Many users tend to be conservative when choosing components, which leads to many bugs that have been fixed still existing in their clusters, allowing uncertainty to spread. In summary, the frontline components of container deployment should use the latest version. The earlier an issue is discovered, the more stable your program will be. In other words, as guinea pigs in the world of open-source, we also need countermeasures; we need to test these components in an automated testing environment.
Many people think that once Kubernetes is installed, the work is done, and they no longer need to worry about the environment. Indeed, containers can solve some operational problems. However, the reliability of the application architecture cannot depend solely on Kubernetes. Why did the concept of SRE (Site Reliability Engineering) come into play in the DevOps field after the introduction of containers? It is because business assurance has always been the core capability of a business and cannot rely on Kubernetes. After using Kubernetes, you need to pay even more attention to the stability of your architecture.
Configuration Testing of kubeadm #
The purpose of kubeadm is to install and upgrade Kubernetes more smoothly. At first, I was resistant to it because the binary installation seemed simpler and easier to troubleshoot. However, as my installation experience grew, I found that binary installation was still not standardized, with many manual configuration steps that couldn’t satisfy the goal of one-click installation. Kubeadm is the only officially recognized installation project, which indicates the community’s recognition of its cloud-native installation and configuration methods.
This leads to the question: does kubeadm always produce the same installation results? No, it doesn’t. It considers single-node mode, high availability mode, mixed-component image mode, and grouped-component image mode. These choices will result in many options and some unknown problems during the installation process. Due to the different versions of kubeadm, its installation process and details will be fine-tuned. Therefore, we should try to use the latest version of kubeadm for installation, so that we can enjoy the benefits of one-click deployment. Many tasks that used to be done manually, such as issuing self-signed certificates, are now automatically handled by kubeadm. Additionally, parameters for environment tuning before installation are also automatically generated. These are all operations that used to require manual consideration and execution. Currently, my experience is: take the official documentation as the basis, choose the latest version of kubeadm as the best component, and then plan the installation of Kubernetes. So far, I have not encountered any difficulties.
Configuration Testing of Calico, IPVS, and kube-dns Components #
Many users are overwhelmed when it comes to container networking because there are too many solutions available, making it difficult for users to make a clear choice. Each solution claims to be the best, but in reality, none of them are perfect. Currently, there is no officially recommended container networking solution. According to the Kubernetes official documentation, networking is not included in the cluster components. This is the community’s choice, which is understandable. However, when we install a cluster, if we don’t configure the networking, Kubernetes cannot host containerized applications.
To solve this practical problem, we need to choose a suitable solution.
Calico is considered the most ideal solution because of its simple configuration. Under the IPIP mode, it creates a container network that performs similarly to the host’s network card, with minimal overhead, even in a scale of 100 physical machines. This was something that previous versions of Calico couldn’t achieve, but the latest version has improved performance. As for kube-dns, it depends on the container network. Only when you have a container network can kube-dns work properly. Currently, CoreDNS is the mainstream option, and it also performs well in a scale of 100 physical machines. So far, there haven’t been any major issues.
IPVS is a kernel module that mainly replaces kube-proxy for north-south traffic. However, its functionality is still limited to above the proxy layer and does not support east-west traffic. To handle east-west traffic, it still relies on iptables. Huawei proposed the IPVS solution because they found performance bottlenecks when using iptables to handle north-south traffic. With the emergence of eBPF programmable packets, the IPVS solution has started to enter a transitional phase. eBPF technology not only supports north-south traffic but also east-west traffic, making it a perfect replacement for iptables. After all, iptables was designed for firewalls, and its complex and dynamically changing rules can have an impact on the system, which is not desirable for users.
In summary, the configuration of these components currently has default settings, and they generally only need to be configured once. As long as they can function properly after being tested, there is no need to spend too much effort on their configuration.
Conclusion #
The so-called configuration testing of Kubernetes relies on users installing it themselves. It is difficult to gain firsthand experience with just examples. However, the purpose of testing is to help users understand the problems and solutions encountered during the installation process. We need to feel the differences of open-source software, which relies on the power of the community to maintain version stability.
During the installation process, everyone will definitely encounter various problems. In addition to verifying and troubleshooting on your own, you can confidently write down your problems in the community’s issue list, facilitating mutual communication. Many users in China do not understand the value of this interaction because the open-source movement is a social activity that relies on user interaction. If no one provides feedback, the software will only get worse.
We need to understand the significance of kubeadm, which is that more configurations may be replaced by default values in the future, and users only need to execute one command to obtain a performance-optimized Kubernetes environment. This will take time, and we can look forward to it.
Reference: