16 Cilium Container Networking Practical Implementation #

As more and more enterprises adopt Kubernetes, the demand for programmable data planes is becoming more widespread due to new requirements such as multi-cloud, security, visibility, and scalability. In addition, new technologies like service meshes and serverless have placed more customization requirements on the underlying Kubernetes infrastructure. These new demands have one thing in common: they require a more programmable data plane that can perform network data operations with Kubernetes awareness without sacrificing performance.

The Cilium project introduces extended Berkeley Packet Filter (eBPF) technology to expose programmable hooks to the network stack in the Linux kernel. This allows network packets to be quickly exchanged contextually without needing to switch back and forth between user and kernel spaces. This is a new network paradigm and the core idea behind the Cilium container networking project.

Why do we need to implement Cilium container networking? #

The development of container networking solutions for Kubernetes has been a competitive space with various offerings. In the past, due to the immaturity of CNI network solutions, many enterprises were hesitant to use them and opted for host networking instead, fearing unacceptable performance impact from container networking. However, as time went on, mainstream container networking solutions like Calico have undergone numerous production environment tests and have been able to meet acceptable network performance metrics in most scenarios. With more successful implementations, enterprises have started to adopt container networking on a larger scale. As the volume of traffic increases, users’ understanding of Kubernetes networking has become more consistent. In general, there are two main types: Cluster IP, a virtual network layer for reverse proxies, and Pod IP, the network data plane for inter-container communication. For the technology implementation of the reverse proxy virtual network, early versions of kube-proxy used iptables, later the introduction of IPVS solved the performance problems of network orchestration in large-scale container clusters. When you look at the overall structure from a high-level perspective, you can see that the Kubernetes network data platform is fragmented and lacks a unified network policy orchestration and isolation system. Obviously, this kind of technical structure also cannot introduce data visualization capabilities. This is where Istio service mesh comes in, by adding an envoy sidecar to achieve network traffic visualization. However, this additional boundary gateway adds another layer of reverse proxy, which slows down network performance. Cilium natively orchestrates network data through eBPF, making visualization simpler.

Cilium’s strength lies in its ability to customize isolation strategies through eBPF, allowing for more container network isolation in non-trusted host environments. This eliminates concerns about data leakage and allows users to focus more on data business connectivity. Because of eBPF’s programmability, we can also add various customized plugins based on business requirements, making the data platform more flexible and secure.

Implementation of Cilium CNI #

Cilium Agent, Cilium CLI Client, and CNI Plugin run on every node in the cluster as daemon sets. The Cilium CNI plugin performs all tasks related to the network pipeline, such as creating link devices (veth pairs), assigning IP addresses to containers, configuring IP addresses, routing tables, sysctl parameters, etc. The Cilium Agent compiles BPF programs and runs them on critical points in the network stack of the kernel.

Cilium provides two networking modes:

• Overlay mode: This is the default networking mode for Cilium. All nodes in the cluster are interconnected with a mesh network using a UDP-based encapsulation protocol. This can be VXLAN (default) or Geneve (Generic Network Virtualization Encapsulation). In this mode, Cilium automatically forms an overlay network without any configuration needed by the user through the --allocate-node-cidrs option in kube-controller-manager.
• Direct/local routing mode: In this configuration, Cilium hands off all packets that do not target another local endpoint to the Linux kernel’s routing subsystem. This setup requires an additional routing daemon such as Bird, Quagga, BGPD, Zebra, etc., which announces non-local node-assigned prefixes to all other nodes via the IP of the node. Compared to VxLAN overlay, the BGP solution has better performance and, more importantly, it enables routing of container IPs without any additional mesh configuration.

Cilium creates three virtual interfaces in the host network space: ciliumhost, ciliumnet, and ciliumvxlan. When the Cilium Agent starts, it creates a veth pair named “ciliumhost -> ciliumnet” and sets the first IP address of the CIDR range as the ciliumhost and as the gateway for the CIDR. The CNI plugin generates BPF rules that are compiled and injected into the kernel to solve connectivity issues between veth pairs. The data link is shown below:

Implementation Practice #

Since Cilium has high kernel requirements, I originally thought it could only be installed on Ubuntu. However, after reviewing the documentation, I found that it is supported on CentOS 7.x and later versions. The installation steps are as follows:

1. Mount the bpf module:

sudo mount bpffs -t bpf /sys/fs/bpf

1. Modify the /etc/fstab file and add the following configuration at the end:

```
bpffs                  /sys/fs/bpf             bpf     defaults 0 0
```

1. Install a clean Kubernetes system. Here, I will use K3s for quick setup:

```bash
export MASTER_IP=$(ip a | grep global | grep -v '10.0.2.15' | awk '{print $2}' | cut -f1 -d '/')
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --no-flannel --node-ip=${MASTER_IP} --node-external-ip=${MASTER_IP} --bind-address=${MASTER_IP} --no-deploy servicelb --no-deploy traefik" sh -
```

1. After this, when you run the following command, you should see the local-path, metrics-server, and coredns pods in the Pending state. This is normal.

```bash
[root@dev-mng-temp ~]# kubectl get po -n kube-system
NAME                                     READY   STATUS    RESTARTS   AGE
coredns-7944c66d8d-xn96v                 0/1     Pending   0          3m2s
local-path-provisioner-6d59f47c7-77bfz   0/1     Pending   0          3m2s
metrics-server-7566d596c8-8bhrq          0/1     Pending   0          3m2s
```

1. Install helm3:

sudo wget — no-check-certificate https://get.helm.sh/helm-v3.2.4-linux-amd64.tar.gz
sudo tar -zxvf helm-v3.2.4-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm

1. Install the Cilium suite:

# sudo helm repo add cilium https://helm.cilium.io/

# sudo helm install cilium cilium/cilium --set global.device=eth0 --set global.tag="v1.8.1" \
--set global.containerRuntime.integration="containerd" \
--set global.containerRuntime.socketPath="/var/run/k3s/containerd/containerd.sock" \
--set global.kubeProxyReplacement="strict" \
--set global.hubble.enabled="true" \
--set global.hubble.listenAddress=":4244" \
--set global.hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
--set global.hubble.relay.enabled="true" \
--set global.hubble.ui.enabled="true" \
--kubeconfig /etc/rancher/k3s/k3s.yaml --namespace kube-system

NAME: cilium
LAST DEPLOYED: Fri Oct 9 19:53:59 2020
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.8.4.

For any further help, visit https://docs.cilium.io/en/v1.8/gettinghelp

1. Remove the IPAM feature:

kubectl edit configmap cilium-config -n kube-system

### Remove the following configurations
  ipam: cluster-pool
  k8s-require-ipv4-pod-cidr: "true"
  k8s-require-ipv6-pod-cidr: "false"

Wait a moment, and you will see that all Cilium-related containers have started. Cilium container network configuration is successful.

1. Change the Hubble UI console service to NodePort mode:

sudo kubectl edit svc hubble-ui -n kube-system

sudo kubectl get svc hubble-ui -n kube-system

You can then view the visualization data platform, as shown in the following image:

Summary #

From the actual experience, Cilium’s network solution can meet the needs of regular container networks. Its visualization console Hubble is the most native implementation of data plane visualization, which is obviously more advanced than Istio’s solution. The data visualization aspect has surprised me a bit. I didn’t expect that eBPF’s programming ability could be so strong, bringing more expectations for more plugin features in the future. Because Cilium technology is still very new, based on practical experience, I recommend that you use it boldly in development and testing environments, but wait a little longer for production environments. I believe that after six months of tempering, Cilium should become the most widely used container network solution in the Kubernetes community.