05 Methodology Proposal for Resolving Practical Issues in Kubernetes (K8s) #

Readers who have experience in implementing technology should understand that any technology, including Kubernetes, will form a set of mature methodologies after a period of accumulation. In the process of implementing Kubernetes, there are prominent issues such as cluster construction, CI/CD pipeline construction, resource tenant management, and security. In order to help newcomers to Kubernetes avoid pitfalls, this article will provide a deep analysis based on the experience of previous practitioners.

Building an Elastic Cluster Strategy #

The Kubernetes cluster architecture is designed for container management in a single data center. During the implementation process in an enterprise, different cluster deployment strategies have evolved due to the changes in scenarios, businesses, and requirements. These strategies can generally be classified into unified shared clusters, independent environment multi-zone clusters, application environment multi-zone clusters, and dedicated small clusters:

Cost	Management	Elasticity	Security
Unified Shared Clusters
Independent Environment Multi-zone Clusters
Application Environment Multi-zone Clusters
Dedicated Small Clusters

Based on the above analysis, the best current approach is to deploy multiple clusters centered around environments or applications in order to achieve the optimal benefits.

Building an Elastic CI/CD Pipeline Strategy #

There are many tools available for building CI/CD pipelines, however, regardless of the tool used, we often struggle with how to introduce Kubernetes into the system. Through practice, it has been found that the combination of GitOps workflow and Kubernetes can bring many benefits. Here, we can refer to the architecture diagram of the well-known CI/CD tool JenkinsX as a reference:

By combining GitOps with Jenkins’ Pipeline, we can create pipelines that meet the needs of business scenarios, allowing applications to switch and iterate continuously in different environments. The advantage of this strategy is that it fully utilizes Git’s version workflow to control the integration quality of the code, and the characteristics of the pipeline allow for full expression of continuous iterative capabilities.

Building an Elastic Multi-Tenant Resource Management Strategy #

The Kubernetes internal account system includes User, Group, and ServiceAccount. After obtaining resource permissions through RBAC authorization, the permission capabilities of these three resources are the same. Due to different use cases, we generally provide User and Group objects for human permissions. When dealing with calls between pods or external system services and the Kubernetes API, ServiceAccounts are usually used. In the native Kubernetes environment, we can bind accounts and resources through Namespaces to achieve API-level multi-tenancy. However, the native multi-tenancy configuration is too cumbersome, so we usually use some auxiliary open source multi-tenancy tools to help us, such as the Kiosk multi-tenancy extension suite:

Through the Kiosk design workflow diagram, we can clearly define the permissions for each user and configure a reasonable resource environment. This simplifies the previously cumbersome configuration process into a default tenant template, making the configuration process of multi-tenancy more standardized.

Building an Elastic Security Strategy #

Based on the security considerations of a Kubernetes container cluster, there are many attack vectors to consider. Therefore, to create a comprehensive security strategy, we still need to refer to security experience at the system level. According to the security framework design of the well-known global security knowledge base MITRE ATT&CK, the following aspects need to be taken into account:

Initial Access #

The main aspect we need to consider is the audit work for authentication and authorization. For example, in cloud-based Kubernetes, if the authentication credentials are leaked, the container cluster will be exposed. For example, the Kubeconfig file, which is the administrative authorization file for cluster administrators, once obtained by an attacker, will expose the entire cluster to the attacker. In addition, potential bugs in the base image, vulnerabilities in applications, and other issues can pose security risks to the cluster if not carefully handled. The built-in open-source Kubernetes Dashboard should also not be exposed on the public network, and its port security needs to be ensured.

Execution #

In this attack surface, we need to prevent attackers from being able to directly execute programs within containers. For example, the kubectl exec command in Kubernetes can be used to enter the container and execute commands. Moreover, if attackers have the permission to run containers, they can use legitimate Service Account credentials to access the API Server and attempt attacks. If the container has an SSH service built-in, attackers can also gain remote access to the container through phishing techniques.

Persistence #

This attack surface primarily utilizes cluster features to deploy backdoors in order to gain continuous control over cluster resources. For example, providing containers with backdoor programs allows hiding backdoor programs on each host. Additionally, Kubernetes clusters by default support hostPath mounting, which allows attackers to mount a writable directory and leave a backdoor program on the host, making it easier to mount this directory and execute the backdoor program in the future using the Cronjob technique.

Privilege Escalation #

Here, the main issue is that containers have the ability to execute with system privileges by default. When containers start with the Privileged parameter, they have direct access to the system capabilities provided by the host’s kernel, enabling attackers to execute system-level attacks. Moreover, Kubernetes has the built-in cluster-admin super-administrator role, and if attackers have the cluster-binding power, they can assign ordinary users the cluster-admin role and directly obtain the role and power of a cluster administrator.

Defense Evasion #

This technique mainly involves attackers hiding their attack traces by clearing logs or events. For example, attackers can hide the destructive behavior of backdoor program containers by deleting container system logs. Additionally, attackers can reset container instances using kubectl delete to effectively clear event logs and hide attack behaviors.

Credential Access #

This attack technique mainly involves attackers exploiting knowledge of Kubernetes features to specifically scan and obtain key credentials. For example, potential attack keys can be obtained by scanning secrets. Additionally, container applications generally assign key locations through environment variables, and attackers can also traverse environment variables to obtain sensitive credential data.

Discovery #

After attackers are familiar with the features of the Kubernetes cluster, they can use scanning techniques to identify necessary attack vulnerabilities through API Server interfaces, Kubelet API interfaces, and Pod ports. Additionally, attackers can run containers in the cluster and then penetrate the Dashboard open-source panel container to collect cluster information using the identity of this panel container.

Lateral Movement #

Attackers gain the ability to attack the Kubernetes cluster through vulnerabilities in third-party systems. For example, if attackers have administrative permissions for the Dashboard, they can execute Trojan vulnerabilities within containers using the exec capability of internal containers. Because Pod networks within the cluster are interconnected, attackers can also freely access any Pod containers of interest.

Impact #

Attackers aim to disrupt the environment by destroying, abusing, and disturbing normal execution behavior. For example, deleting Deployment configurations, storage, and computing resources can disrupt container operations. Another example is the illegal abuse of computing resources by running mining programs within containers. Denial of service attacks on the API Server can also make the cluster unavailable.

To address these issues, our security strategy is to provide users with the minimum authorization to run containers. Many users create dedicated management panels to isolate user interaction with Kubernetes, which is a common practice. However, many cloud-based Kubernetes services still expose users to host-level entry points, making security issues exposed to potential attackers. Generally, access to the cluster is restricted to internal personnel only through VPC, but internal security audits still require long-term maintenance. It requires professional security personnel to develop comprehensive prevention strategies to reduce attack risks.

Conclusion #

Through the analysis and practice of the above common issues, we find that the advantages and disadvantages of Kubernetes are very distinct, resulting in mixed feelings. Because Kubernetes is a complex system, many operations become complicated. Therefore, while combining previous experience, many development teams provide open-source enhancement components to strengthen various aspects of Kubernetes clusters. I suggest that everyone should accumulate and use these enhancement components to strengthen their clusters, so that we can make good use of Kubernetes clusters by standing on the shoulders of giants.