31 Network Communication Cni What's the Deal and How It Works

31 Network Communication CNI What’s the Deal and How It Works #

Hello, I am Chrono.

By now, we are already very familiar with Kubernetes, which is a cluster operating system that can manage a large number of compute nodes and the applications running inside them. However, there is an important fundamental concept we have yet to learn, and that is “network communication.”

Back in Lesson 6 of the “Getting Started” section, we briefly introduced Docker’s network modes, and then in Lesson 17 of the “Intermediate” section, we installed a network plugin called Flannel for Kubernetes. These are all related to networking, but we only scratched the surface and didn’t delve too deep.

If you are someone who likes to dig deep into things, you may be curious: How does Flannel actually work? Why does it enable normal communication within the Kubernetes cluster? Are there any other network plugins?

Today, let’s talk about this topic and discuss the Kubernetes Container Network Interface (CNI) standard, as well as high-performance network plugins like Calico and Cilium.

Kubernetes Networking Model #

Before we delve into Kubernetes networking, let’s briefly review Docker’s networking knowledge.

Do you still remember Docker’s null, host, and bridge network modes? Here, I have redrawn a diagram that describes the most commonly used bridge network mode in Docker:

Docker creates a bridge called “docker0” by default, with the private subnet “172.17.0.0/16”. Each container creates a pair of virtual network interfaces (veth pair), with one interface “plugged” into the container and the other into the bridge. This allows containers to communicate with each other.

Docker’s network solution is simple and effective, but the problem is that it only works within a single host environment. Communication across hosts is very challenging and requires port mapping and network address translation (NAT).

To address Docker’s network limitations, Kubernetes proposes its own networking model called “IP-per-pod,” which can better meet the networking requirements of a cluster system. It is based on the following four basic assumptions:

Each pod in the cluster has a unique IP address.
All containers within a pod share the same IP address.
All pods in the cluster belong to the same subnet.
Pods can directly access other pods using their IP addresses without the need for troublesome network address translation (NAT).

I have created a diagram illustrating the Kubernetes networking model. Take a look:

This network model liberates pods from the constraints of the host and presents a “flat” network model that is easy to understand and communicate with.

Because pods have their own unique IP addresses, similar to virtual machines, and can communicate directly with one another, it becomes easy to implement tasks such as domain name resolution, load balancing, and service discovery. Previous sysadmin experience can be directly applied, making it very user-friendly for application management and migration.

What is CNI #

The network model defined by Kubernetes is perfect, but implementing this model is not easy. Therefore, Kubernetes has specifically created a standard: CNI (Container Networking Interface).

CNI defines a series of common interfaces for network plugins. Developers only need to follow this specification to integrate with Kubernetes, create virtual network cards, allocate IP addresses, set routing rules for Pods, and ultimately achieve the “IP-per-pod” network model.

Based on different implementation technologies, CNI plugins can be roughly divided into three types: “Overlay,” “Route,” and “Underlay.”

The term “Overlay” refers to building a “logical network” that works on top of the underlying real network. It encapsulates the original Pod network data, sends it through the underlying network, and then unpacks it at the destination. Because of this characteristic, it has low requirements for the underlying network and strong adaptability, but it has additional transmission costs and lower performance.

The “Route” mode also works on top of the underlying network, but it does not encapsulate and unpack data. Instead, it uses the system’s built-in routing functions to enable communication between Pods across hosts. Its advantage is high performance, but it has a strong dependency on the underlying network. If the underlying network does not support it, it cannot work.

The “Underlay” mode directly uses the underlying network to implement CNI, which means that Pods and host machines are in the same network, and Pods and host machines are equal. It has the strongest dependence on the underlying hardware and network, so it is not very flexible, but it has the highest performance.

Since the release of CNI in 2015, due to its loosely defined interface, there has been a large space for creativity in the community, and many network plugins have emerged. Flannel mentioned in [Lesson 17] is one of them.

Flannel (https://github.com/flannel-io/flannel/) was developed by CoreOS (acquired by Redhat). It was originally an Overlay mode network plugin that used UDP and VXLAN technology. Later, it supported the Route mode through Host-Gateway technology. Flannel is simple and easy to use. It is the most popular CNI plugin in Kubernetes, but it does not perform well in terms of performance, so it is generally not recommended for production environments.

There are currently two other commonly used CNI plugins: Calico and Cilium, let’s give a brief introduction.

Calico (https://github.com/projectcalico/calico) is a Route mode network plugin that uses the Border Gateway Protocol (BGP) to maintain routing information. It has better performance than Flannel and supports multiple network policies, such as data encryption, security isolation, and traffic shaping.

Cilium (https://github.com/cilium/cilium) is a relatively new network plugin that supports both Overlay and Route modes. Its feature is the deep use of Linux eBPF technology to operate network data at the kernel level, so it has high performance and can flexibly implement various functions. In 2021, it joined CNCF as an incubation project and is a very promising CNI plugin.

How does the CNI plugin work? #

Let’s start with Flannel, which is relatively simple, and use it as an example to see how CNI works in Kubernetes.

It is important to note that computer networks are complex, with many concepts such as IP addresses, MAC addresses, subnets, network interfaces, bridges, and routing. Data flows through multiple devices, and it can be complicated to understand the overall picture. Today, we will provide a general description without going into too much detail about the lower-level aspects.

Let’s start by creating three Nginx Pods using Deployment in our test environment:

kubectl create deploy ngx-dep --image=nginx:alpine --replicas=3

Using the command kubectl get pod, we can see that two Pods are running on the master node with IP addresses “10.10.0.3” and “10.10.0.4”, and another Pod is running on the worker node with the IP address “10.10.1.77”:

Flannel uses the VXLAN-based overlay mode by default. I have created a diagram to illustrate the network structure of the entire cluster, which you can compare to Docker’s network structure:

From a single-machine perspective, the network structure of Flannel is almost identical to that of Docker, except that the bridge is named “cni0” instead of “docker0”.

Next, let’s perform some operations to see how the virtual network interface in the Pod is connected to the cni0 bridge.

By executing the command ip addr inside the Pod, you can see the virtual network interface “eth0”:

You should pay attention to its format. The first number “3” is the index, indicating that it is the 3rd device. “@if45” represents the other end of the connection, and the index is 45.

Since the host of this Pod is the master node, we need to log in to the master node to see the network situation. We can use the command ip addr again:

Here you can see the 45th device on the host (master) node, named veth41586979@if3. “veth” indicates that it is a virtual network interface, and “@if3” corresponds to the 3rd device in the Pod, which is the “eth0” network interface.

So, how can we view the information of the “cni0” bridge? We need to use the command brctl show on the host (master) machine:

From this screenshot, you can find that there are four virtual network interfaces on the “cni0” bridge, and the third one is “veth41586979”. Therefore, this virtual network interface is “plugged” into the “cni0” bridge, and then, due to the paired property of virtual network interfaces, the Pod is connected to the “cni0” bridge.

It is not easy to understand the relationship between network interfaces and bridges using only Linux commands. Therefore, I have integrated them into the following diagram, with dotted lines to indicate the connections. This way, you can better understand the references between the Pod, veth, and cni0:

Using the same method, you can find out that the network interface of another Pod “10.10.0.4” is veth2b3ef56d@if3, and it is also connected to the “cni0” bridge. Therefore, with the help of this bridge, the Pods on this machine can communicate with each other directly.

After understanding the local network, let’s take a look at the network between hosts. The key is the routing table of the nodes, which can be viewed using the command route:

It tells us the following information:

Data in the 10.10.0.0/24 subnet should be sent through the cni0 device, which is the “cni0” bridge.
Data in the 10.10.1.0/24 subnet should be sent through the flannel.1 device, which is Flannel.
Data in the 192.168.10.0/24 subnet should be sent through the ens160 device, which is the network card of our host machine.

Assuming we want to access “10.10.1.77” on the worker node from the “10.10.0.3” on the master node, since the “cni0” bridge on the master node only manages the “10.10.0.0/24” subnet, according to the routing table, any traffic destined for “10.10.1.0/24” should be handled by flannel.1. This triggers the working process of the Flannel plugin.

Flannel will then determine how to send the data to the other node by querying various tables. Since this process can be tedious, I won’t go into detail, but you can refer to the diagram below, which uses commands like ip neighbor and bridge fdb:

The result obtained by Flannel is that the data should be sent to “192.168.10.220”, which is the worker node. It will then encapsulate the original network packet with additional information, such as VXLAN headers, and send it out through the “ens160” network card. When the worker node receives the packet, it will perform the reverse process and deliver the data to the target Pod.

Using Calico Network Plugin #

At this point, you may find Flannel’s overlay processing quite complicated and confusing. So, let’s take a look at another routing mode plugin called Calico.

You can find the installation method for Calico on the Calico website (https://www.tigera.io/project-calico/). I have chosen the “Self-managed on-premises” option, and you can directly download the YAML file:

wget https://projectcalico.docs.tigera.io/manifests/calico.yaml

Since Calico uses large images, you can consider pulling the images using docker pull on each node to speed up the installation:

docker pull calico/cni:v3.23.1
docker pull calico/node:v3.23.1
docker pull calico/kube-controllers:v3.23.1

Installing Calico is very simple; you just need to use kubectl apply (remember to remove Flannel before installing):

kubectl apply -f calico.yaml

After installation, let’s check the running status of Calico. Please note that it is also in the “kube-system” namespace:

Next, let’s create three Nginx Pods for our experiments:

kubectl create deploy ngx-dep --image=nginx:alpine --replicas=3

We will see that the master node has two Pods, and the worker node has one Pod. However, their IP addresses are different from the ones we saw with Flannel. They are now “10.10.219.” and “10.10.171.” respectively. This indicates that Calico has a different IP address allocation strategy than Flannel:

Now let’s take a look at the network interfaces inside the Pods. You will notice that although there are still virtual interfaces, the name of the interface on the host has changed to calica17a7ab6ab@if4, and it is not connected to the “cni0” bridge:

That seems strange, doesn’t it?

In fact, this is a normal behavior caused by Calico’s working mode. Since Calico is not an overlay mode but a routing mode, it does not use the Flannel system. Instead, it creates routing rules on the host to allow packets to “jump” directly to the destination interface without passing through the bridge.

Let’s take a look at the routing table on the node, and it will become clear:

Suppose Pod A with IP address “10.10.219.67” wants to access Pod B with IP address “10.10.219.68”. By checking the routing table, the packet knows it needs to go through the device “cali051dd144e34”, which happens to be inside Pod B. Therefore, the data will go directly into Pod B’s network interface, bypassing the bridge intermediary step.

I have also created an architecture diagram for Calico’s network, so you can compare it with Flannel for better understanding:

As for how cross-host communication is routed in Calico, you can follow the routing table step by step to “jump” to the target Pod (hint: using the tunl0 device).

Summary #

After discussing so much, you should have seen that the whole network data transmission process of Kubernetes involves many details, and many aspects are involved in it. It is not easy to thoroughly understand it.

Fortunately, CNI solves these problems by using the principle of “inversion of control” and delegates these tasks to plugins. Regardless of the underlying environment or the implementation of the plugin, we will have a clean and tidy network space in the Kubernetes cluster.

Let me summarize today’s content briefly:

Kubernetes uses the “IP-per-pod” network model, where each Pod has a unique IP address, making it easy to manage.
CNI is the network plugin interface standard defined by Kubernetes, and it can be categorized into “Overlay,” “Route,” and “Underlay” based on its implementation. Common CNI plugins include Flannel, Calico, and Cilium.
Flannel supports the Overlay mode. It uses the cni0 bridge and the flannel.1 device. Local communication goes directly through cni0, while cross-host communication encapsulates the original packets into VXLAN packets and sends them over the host network card, resulting in a performance loss.
Calico supports the Route mode. It does not use the cni0 bridge. Instead, it creates routing rules and sends packets directly to the destination network card, resulting in higher performance.

Homework #

Now it’s time for your homework. I have prepared two questions for you to think about:

Kubernetes does not have built-in network implementation, but instead uses CNI to define a standard interface. What are the advantages of doing so?
What are your thoughts on the working models of the Flannel and Calico network plugins?

Feel free to participate in the discussion by leaving your comments. This is the final lesson of our knowledge study. In the next lesson, we will review everything we have learned. The dawn is just ahead, and I look forward to seeing your growth in the upcoming hands-on and video lessons. See you in the next lesson.