10 Practical Implementation of East-West Traffic Component Calico #
Kubernetes does not have a native networking solution, so it presents us with a choice. Which networking solution is the best? The networking problem has always confused users in the community, to the point where a multitude of solutions emerged for different scenarios in the early days. One of the better solutions is the network component Calico, which we will introduce today. It is important to note that Calico is not the only solution. In the community, we can still see many excellent solutions such as Cilium, OvS, Contiv, Flannel, etc. However, most of the current domestic industry solutions are based on the practice of using Calico as the main solution. Introducing Calico can serve as a case demonstration.
Principles of Container Network Routing #
It is well known that the native container networking model is based on the veth virtual bridge of a single machine and cannot interconnect across hosts. To enable interconnection between containers across hosts, the following three points need to be supported:
- The network control plane needs to ensure the uniqueness of container IPs.
- Two containers need to be placed on the same data plane.
- Tools are needed to automatically resolve container network address translation.
Here, we use an example of native network routing to help everyone understand the basic principles of container network interconnection:
Figure: Example of the Docker 19.03.12 version’s direct routing mode
Configure docker0 on Host 1 and Host 2 separately, and restart the Docker service to take effect #
Edit the /etc/docker/daemon.json file on Host 1 and add the content: "bip": "ip/netmask".
{
"bip": "172.17.1.252/24"
}
Edit the /etc/docker/daemon.json file on Host 2 and add the content: "bip" : "ip/netmask".
{
"bip": "172.17.2.252/24"
}
Execute the following command on both Host 1 and Host 2 to restart the Docker service to make the modified docker0 subnet take effect.
systemctl restart docker
Add routing rules #
Add the following routing rules on Host 1:
route add -net 172.17.2.0/24 gw 172.26.15.215
Add the following routing rules on Host 2:
route add -net 172.17.1.0/24 gw 172.26.14.120
In theory, after configuring the routing, it should be possible to connect to the docker0 on Host 2 from Host 1. In actual scenarios, however, the switch will check the MAC address of docker0 and directly drop packets from this illegal address, making it impossible for the containers across hosts to communicate. But this doesn’t hinder our understanding of the principle. The Calico network routing principle is consistent with the static routing principle demonstrated above. So how does Calico achieve connectivity?
After installing Calico, check on the host as follows:
On Host 1 #
vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UNKNOWN group default link/ether 66:18:b6:89:bd:6f brd ff:ff:ff:ff:ff:ff inet 192.168.206.0/32 brd 192.168.206.0 scope global vxlan.calico
On Host 2 #
vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UNKNOWN group default link/ether 66:85:17:26:23:b1 brd ff:ff:ff:ff:ff:ff inet 192.168.49.0/32 brd 192.168.49.0 scope global vxlan.calico
By comparing the above content, we can see that Calico also assigns a fixed IP range to each host to ensure that the container network IPs do not conflict. Let’s take a look at the routing rules:
root@ip-172-26-8-126:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.26.0.1 0.0.0.0 UG 100 0 0 eth0
172.26.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
172.26.0.1 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
192.168.49.1 0.0.0.0 255.255.255.255 UH 0 0 0 cali3f1b566e6d6
192.168.49.2 0.0.0.0 255.255.255.255 UH 0 0 0 cali408c6db5188
192.168.206.0 172.26.5.8 255.255.255.192 UG 0 0 0 eth0
root@ip-172-26-5-8:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.26.0.1 0.0.0.0 UG 100 0 0 eth0
172.26.0.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
172.26.0.1 0.0.0.0 255.255.255.255 UH 100 0 0 eth0
192.168.49.0 172.26.8.126 255.255.255.192 UG 0 0 0 eth0
192.168.206.1 0.0.0.0 255.255.255.255 UH 0 0 0 cali517a7f7f853
192.168.206.3 0.0.0.0 255.255.255.255 UH 0 0 0 cali8d8ae1f64d9
192.168.206.4 0.0.0.0 255.255.255.255 UH 0 0 0 cali99a5d6c4e2d
192.168.206.5 0.0.0.0 255.255.255.255 UH 0 0 0 cali3b6d130f2eb
192.168.206.6 0.0.0.0 255.255.255.255 UH 0 0 0 calid8aeffc724c
Both host’s routing tables have routing information for each other’s IP.
We tested network connectivity by deploying containers:
root@ip-172-26-5-8:~# cat pingtest-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: pingtest-deployment
labels:
app: pingtest
spec:
replicas: 3
selector:
matchLabels:
app: pingtest
template:
metadata:
labels:
app: pingtest
spec:
containers:
- name: pingtest
image: busybox
args: ["sleep", "infinity"]
root@ip-172-26-5-8:~# kubectl apply -f pingtest-deployment.yaml
root@ip-172-26-5-8:~# kubectl get pod -l app=pingtest -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pingtest-deployment-6dcb8d6c77-8ntqf 1/1 Running 0 10m 192.168.49.4 ip-172-26-8-126 <none> <none>
pingtest-deployment-6dcb8d6c77-l5hq2 1/1 Running 0 10m 192.168.49.5 ip-172-26-8-126 <none> <none>
pingtest-deployment-6dcb8d6c77-6fcdn 1/1 Running 0 6m48s 192.168.206.7 ip-172-26-5-8 <none> <none>
Because the latest default configuration of Calico is in vxlanMode, and you do not have permission to modify the router, you need to change the ipipMode to Always.
root@ip-172-26-12-198:~# cat pool.json
{
"kind": "IPPoolList",
"apiVersion": "projectcalico.org/v3",
"metadata": {
"resourceVersion": "2306"
},
"items": [
{
"kind": "IPPool",
"apiVersion": "projectcalico.org/v3",
"metadata": {
"name": "default-ipv4-ippool",
"uid": "0ba1e107-0582-4b7b-b99f-f7105525e987",
"resourceVersion": "763",
"creationTimestamp": "2020-08-10T16:59:28Z"
},
"spec": {
"cidr": "192.168.0.0/16",
"vxlanMode": "Never",
"ipipMode": "Always",
"natOutgoing": true,
"blockSize": 26,
"nodeSelector": "all()"
}
}
]
}
After modifying it, you can check the routing rules in the command line to confirm that the virtual network is using the tunnel interface:
root@ip-172-26-12-198:~# ip route
default via 172.26.0.1 dev eth0 proto dhcp src 172.26.12.198 metric 100
172.26.0.0/20 dev eth0 proto kernel scope link src 172.26.12.198
172.26.0.1 dev eth0 proto dhcp scope link src 172.26.12.198 metric 100
192.168.31.0 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.31.0/26 via 172.26.5.10 dev eth0 proto 80 onlink
192.168.31.1 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.31.3 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.31.4 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.31.5 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.31.7 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.31.64/26 via 172.26.5.10 dev tunl0 proto bird onlink
192.168.41.137 dev calie486eacd845 scope link
192.168.41.138 dev calif383cce9723 scope link
192.168.41.139 dev calia4f3d6b96e0 scope link
192.168.41.140 dev cali391d55f6fc3 scope link
192.168.41.141 dev cali981dc37e1ca scope link
192.168.41.142 dev calic0a72d40721 scope link
192.168.41.143 dev calicfb8f80c8c5 scope link
blackhole 192.168.41.192/26 proto bird
192.168.41.193 dev calia6f4721616e scope link
You can use kubectl exec to enter the Pod container for connectivity testing:
root@ip-172-26-12-198:~# kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pingtest-deployment-6dcb8d6c77-95vrw 1/1 Running 0 23m 192.168.41.139 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-p4cqx 1/1 Running 0 23m 192.168.41.140 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-kfmhp 1/1 Running 0 23m 192.168.41.137 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-w582x 1/1 Running 0 23m 192.168.41.141 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-ts9fh 1/1 Running 0 23m 192.168.41.138 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-ppt2n 1/1 Running 0 22m 192.168.41.142 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-qw46c 1/1 Running 0 22m 192.168.41.143 ip-172-26-12-198 <none> <none>
pingtest-deployment-6dcb8d6c77-972zw 1/1 Running 0 22m 192.168.31.7 ip-172-26-5-10 <none> <none>
root@ip-172-26-12-198:~# kubectl exec -it pingtest-deployment-6dcb8d6c77-972zw -- sh
/ # ping 192.168.41.138
PING 192.168.41.138 (192.168.41.138): 56 data bytes
64 bytes from 192.168.41.138: seq=0 ttl=62 time=0.449 ms
64 bytes from 192.168.41.138: seq=1 ttl=62 time=0.501 ms
Performance of Calico Network #
sh-4.4# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.26.5.10, port 53246
[ 5] local 192.168.31.68 port 5201 connected to 172.26.5.10 port 53248
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.42 GBytes 20.8 Gbits/sec
[ 5] 1.00-2.00 sec 3.07 GBytes 26.4 Gbits/sec
[ 5] 2.00-3.00 sec 2.83 GBytes 24.3 Gbits/sec
[ 5] 3.00-4.00 sec 3.05 GBytes 26.2 Gbits/sec
[ 5] 4.00-5.00 sec 3.12 GBytes 26.8 Gbits/sec
[ 5] 5.00-6.00 sec 2.87 GBytes 24.7 Gbits/sec
[ 5] 6.00-7.00 sec 3.02 GBytes 26.0 Gbits/sec
[ 5] 7.00-8.00 sec 3.04 GBytes 26.1 Gbits/sec
[ 5] 8.00-9.00 sec 3.08 GBytes 26.5 Gbits/sec
[ 5] 9.00-10.00 sec 2.93 GBytes 25.2 Gbits/sec
[ 5] 10.00-10.04 sec 104 MBytes 24.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 29.5 GBytes 25.3 Gbits/sec receiver
In scenarios with less than 10 machines, the efficiency of container transmission depends on the performance of the host network card, and the results show that the performance is good.
Summary #
Calico is a commonly used solution in the industry, and its advantage is its flexible configuration. Because it supports the BGP protocol, it can achieve interconnection between data centers. From a practical point of view, it has the ability to flexibly configure complex scenarios, so it is widely recommended in the industry. However, our goal here is not to recommend Calico. We still need to plan based on the specific needs of your current cluster to make container networking more convenient.
References: