12 Practice Chapter- Kubernetes (K8s) Cluster Configuration Testing #
The content of the second part focuses on the installation and configuration of the core components of Kubernetes. The most popular configuration for cluster components currently includes: containerd, kubeadm, IPVS, Calico, and kube-dns. By referring to the official documentation, readers can independently configure a cluster. However, I have noticed that due to the complexity of cluster configuration, the resulting environments can vary greatly, making it difficult to achieve a unified understanding. The purpose of this test is to verify the knowledge we have learned together and build a complete cluster from scratch to validate our understanding of cluster concepts.
Environment #
Since the first day we started learning container technology, the problem we wanted to solve was environment dependencies. Docker is designed to package environments and applications into images for distribution. When configuring a Kubernetes cluster, the smallest unit we operate on is a Pod, which can be understood as a group of containers. However, this group of containers is not simply putting a set of containers together. The ingenious design of Kubernetes is to use a base container, centered around the “pause” container, to encapsulate all the environment dependencies of the related applications within its runtime. Other related business containers are only added to this runtime, so any issues with these business containers will not break the environment. This is the core design of Kubernetes for building business clusters and effectively solving the availability problem of application services.
Now, it’s time to choose the version of the operating system. You will find that there is no official documentation specifying which version to use. In fact, the official documentation does not have such a convention. This is because the purpose of containers is to solve environment dependencies. However, after so many years of evolution, to be more precise, there is still a core dependency that cannot be solved, which is the Kernel dependency. The features of the Kernel will determine the features of the container. Generally, we refer to the version of Docker when making the choice, with popular versions being 18.09 and 19.03, for example. Notice that you cannot guarantee that these Docker versions will not have issues in specific environments. This is a pitfall we set for ourselves when configuring production environments.
If you are using it within an enterprise, the best approach is to establish a baseline and clearly define the version number. Then, invest manpower to maintain the stability of this version based on extensive practice. Because container technology is developing rapidly, now Kubernetes and Docker are increasingly decoupling, and they are both using containerd to support the management of underlying container runtimes. As users, we cannot avoid this. Here, another problem arises: which version should I choose due to the component changes? Are they stable? Since Kubernetes is a software driven by the open-source community, we must follow the open-source way to use these software in order to gain the correct experience.
Based on my summarized experience, here are some suggestions for your reference: 1. x86-64 is still the best system architecture for containers at present. The mainstream systems are currently focused on the redhat/centos 7.x series and the Ubuntu 16.04 series. For the Red Hat-based kernel, it is mainly above version 3.10, while Ubuntu can reach version 4.4 and above. Some users upgrade the Red Hat-based kernel to version 4.4 through open source kernel repositories, which is also common. The cost of upgrading the kernel is introducing many unknown modules, which makes the system unstable. The ARM system architecture imposes compatibility requirements on the file format of the entire Kubernetes components. When choosing compatibility, it is important to pay attention to whether the corresponding components of Kubernetes are prepared. To summarize, the mainstream operating systems are mainly the Red Hat 7.x series and the Ubuntu LTS 16.04 series. Upgrading the major version of the operating system requires a lot of adaptation work for Kubernetes, which is currently unlikely to be done by the open source community. Pay attention to this.
2. Kubernetes has rapid version updates. The entire community maintains three mainline versions, currently mainly 1.16.x, 1.17.x, and 1.18.x. Each version is iterated approximately every two weeks, mainly to fix bugs. Many teams have summarized some tips for usage, such as using odd or even versions as their main versions, to avoid the instability of the latest versions. This does not mean that odd versions are good or even versions are stable, it is purely a guess. As an open source software, its quality is maintained by the community, and when it comes to users, we are all guinea pigs and need to test the reliability of the components in our own environments. To summarize, for mainstream environments, it is advisable to choose a version that is 1 or 2 sub-versions lower than the latest version as the cycle for maintaining your own software. Maintaining open source software is not free, it requires everyone’s efforts to ensure the reliability of the components in use.
-
In addition to choosing 1 to 2 versions after the main version of Kubernetes, for other related components such as Calico, kube-dns, Containerd, etc., it is recommended to choose the latest version. The main reason is that they are components that run on the front line and are called more frequently, so there are more opportunities to discover problems. The sooner problems are discovered, the faster they can be fixed. This is a principle in the open source field: the earlier a problem is discovered and fixed, the more stable the component becomes. Many users tend to be conservative in component selection, resulting in many bugs that have been fixed still existing in your cluster, allowing the uncertainty to spread. To summarize, the front-line components of running containers should use the latest version. The earlier problems are discovered, the more stable your program will be. In other words, as guinea pigs in the open source world, we should also have countermeasures and test these components through automated testing environments.
-
Many people think that once Kubernetes is installed, the environment will be worry-free. Indeed, containers can solve some of the problems in operations and maintenance. However, the reliability of application architecture cannot rely on Kubernetes alone. Why, after the introduction of containers, has the concept of Site Reliability Engineering (SRE) been introduced in the DevOps field? It means that business guarantee has always been a core competency and cannot rely on Kubernetes. After using Kubernetes, you need to pay more attention to the stability of your architecture.
kubeadm Configuration Test #
The original intention of kubeadm was to install and upgrade Kubernetes in a smoother way. I was initially resistant to it because the binary installation method seemed simpler and more convenient for troubleshooting. However, as I gained more installation experience, I found that the binary installation method could not be standardized, and there were many manual configurations that could not achieve the goal of one-click installation.
kubeadm is the only installation project recognized by the official Kubernetes, which indicates the community’s recognition of its cloud-native installation and configuration methods. The question here is whether kubeadm always produces the same installation results?
No, it takes into account single-node mode, high-availability mode, component mixed image mode, and component grouped image mode. Users will encounter many choices during installation, which will also create some unknown problems. Because kubeadm’s versions are different, its installation process and details will be fine-tuned. We should try to use the latest version of kubeadm for installation, so that we can enjoy the benefits of one-click deployment. Many tasks that used to be done manually, such as issuing self-signed certificates, are now automatically handled by kubeadm. Additionally, parameters for pre-installation environment tuning will also be automatically generated. These are operations that needed to be manually considered and executed before. Currently, the author’s experience is that: based on the official documentation, the latest version of kubeadm is chosen as the best component for Kubernetes installation planning, and so far no difficulties have been encountered.
Configuration Testing for Calico, IPVS, and kube-dns Components #
Many users feel overwhelmed when it comes to container networking because there have been too many solutions in the past, making it difficult for users to make clear choices. Each solution claims to be the best, but none of them are perfect.
Currently, there is no officially recommended container networking solution. According to the Kubernetes official documentation, networking is not included in the cluster components. This is the choice made by the community, and it is understandable. However, when installing a cluster, if the networking is not configured, Kubernetes cannot host business containers. To address this practical issue, we need to choose a suitable solution.
The main reason why Calico is considered the most ideal solution is its simple configuration. In a scenario where a container network is created with the IPIP mode on 100 physical machines, Calico’s performance is almost equivalent to that of the host network card, with very little overhead. This was a problem that previous versions of Calico couldn’t solve, but in the latest version, performance has improved significantly.
As for kube-dns, it relies on the container network. Only when you have a container network can kube-dns work properly. CoreDNS is currently mainstream and performs well on 100 physical machines. There haven’t been any major issues encountered in its operation so far.
IPVS is a kernel module that primarily replaces kube-proxy for north-south traffic. However, its functionality is still limited to the proxy layer, and it does not support kube-proxy’s east-west traffic, which still relies on iptables. Huawei proposed the IPVS solution after discovering performance bottlenecks when using iptables to handle north-south traffic. With the emergence of eBPF programmable packet technology, IPVS is entering a transitional phase. eBPF technology not only supports north-south traffic but also supports east-west traffic, making it a perfect replacement for iptables. After all, iptables is designed for firewalls, and complex and dynamically changing rules can impact the system, resulting in a negative effect on business, which users do not want to see.
To sum up, the configuration of these components currently has default settings. In general, you only need to configure them once and verify that they work properly. There is no need to spend too much effort on the configuration of these components.
Summary #
The so-called configuration testing for Kubernetes relies on hands-on installation experience. It is difficult to gain firsthand experience just by following examples. However, the purpose of testing is to help you understand the problems and solutions encountered during the installation process. We need to feel the differences of open-source software, which relies on the strength of the community to maintain version stability. When installing, you will definitely encounter various issues. In addition to validating and troubleshooting on your own, you can confidently write down your own problems in the community’s issue tracker, facilitating communication among users. Many users in China do not understand the value of this interaction because the open-source movement is a social activity that depends on user interaction. If no one provides feedback, the software will only become worse.
We need to understand the significance of kubeadm, which is that more configurations may be replaced by default values in the future, and users will only need to run a single command to obtain a high-performance Kubernetes environment. This takes time, and we can look forward to it.
Reference: