From 702636b4553a5de3d467e758453b32cfde7d8192 Mon Sep 17 00:00:00 2001 From: Antonin Bas Date: Wed, 9 Sep 2020 14:32:35 -0700 Subject: [PATCH] Add cookbook on how to use Antrea with Multus We add documentation to show how Antrea can be used with Multus: Antrea is used as the default CNI plugin and an "arbitrary" plugin (in our case, macvlan) can be used to attach additional interfaces to designated Pods. Nothing is required on the Antrea side to make it work, so this is just to show how it can be used in practice. Fixes #368 --- docs/cookbooks/multus/README.md | 326 +++ .../multus/assets/testbed-multus-macvlan.svg | 1931 +++++++++++++++++ .../multus/build/cni-dhcp-daemon/Dockerfile | 19 + .../multus/build/cni-dhcp-daemon/README.md | 16 + .../multus/resources/dhcp-daemon.yml | 52 + .../multus/resources/dhcp-server.yml | 58 + .../multus/resources/macvlan-host-init.yml | 62 + .../multus/resources/netAttachDef.yml | 14 + docs/cookbooks/multus/resources/test.yml | 27 + docs/cookbooks/multus/test/Vagrantfile | 64 + test/e2e/infra/vagrant/Vagrantfile | 9 +- 11 files changed, 2577 insertions(+), 1 deletion(-) create mode 100644 docs/cookbooks/multus/README.md create mode 100644 docs/cookbooks/multus/assets/testbed-multus-macvlan.svg create mode 100644 docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile create mode 100644 docs/cookbooks/multus/build/cni-dhcp-daemon/README.md create mode 100644 docs/cookbooks/multus/resources/dhcp-daemon.yml create mode 100644 docs/cookbooks/multus/resources/dhcp-server.yml create mode 100644 docs/cookbooks/multus/resources/macvlan-host-init.yml create mode 100644 docs/cookbooks/multus/resources/netAttachDef.yml create mode 100644 docs/cookbooks/multus/resources/test.yml create mode 100644 docs/cookbooks/multus/test/Vagrantfile diff --git a/docs/cookbooks/multus/README.md b/docs/cookbooks/multus/README.md new file mode 100644 index 00000000000..480806690f5 --- /dev/null +++ b/docs/cookbooks/multus/README.md @@ -0,0 +1,326 @@ +# Using Antrea with Multus + +This guide will describe how to use Project Antrea with +[Multus](https://github.com/intel/multus-cni), in order to attach multiple +network interfaces to Pods. In this scenario, Antrea is used for the default +network, i.e. it is the CNI plugin which provisions the "primary" network +interface ("eth0") for each Pod. For the sake of this guide, we will use the +[macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan) +CNI plugin to provision secondary network interfaces for selected Pods, but +similar steps should apply for other plugins as well, +e.g. [ipvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/ipvlan). + +## Prerequisites + +The only prerequisites are: + * a K8s cluster (Linux Nodes) running a K8s version supported by Antrea. At the + time of writing, we recommend version 1.16 or later. + * [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) + +All the required software will be deployed using YAML manifests, and the +corresponding container images will be downloaded from public registries. + +macvlan requires the network to be able to handle "promiscuous mode", as the +same physical interface / virtual adapter ends up being assigned multiple MAC +addresses. When using a virtual network for the Nodes, some configuration +changes are usually required, which depend on the virtualization technology. For +example: + * when using VirtualBox and [Internal + Networking](https://www.virtualbox.org/manual/ch06.html#network_internal), + set the `Promiscuous Mode` to `Allow All` + * when using VMware Fusion, enable "promiscuous mode" in the guest (Node) for + the appropriate interface (e.g. using `ifconfig`); this may prompt for your + password on the host unless you uncheck `Require authentication to enter + promiscuous mode` in the Network Preferences + +This needs to be done for every Node VM, so it's best if you can automate this +when provisioning your VMs. + +### Suggested test cluster + +If you need to create a K8s cluster to test this guide, we suggest you create +one by following [these +steps](https://github.com/vmware-tanzu/antrea/tree/master/test/e2e#creating-the-test-kubernetes-cluster-with-vagrant). You +will need to use a slightly modified Vagrantfile, which you can find +[here](test/Vagrantfile). Note that this Vagrantfile will create 3 VMs on your +machine, and each VM will be allocated 2GB of memory, so make sure you have +enough memory available. You can create the cluster with the following steps: + +```bash +git clone https://github.com/vmware-tanzu/antrea.git +cd antrea +cp docs/cookbooks/multus/test/Vagrantfile test/e2e/infra/vagrant/ +cd test/e2e/infra/vagrant +./infra/vagrant/provision.sh +``` + +The last command will take around 10 to 15 minutes to complete. After that, your +cluster is ready and you can set the `KUBECONFIG` environment variable in order +to use `kubectl`: + +```bash +export KUBECONFIG=`pwd`/infra/vagrant/playbook/kube/config +kubectl cluster-info +``` + +The cluster that you have created by following these steps is the one we will +use as an example in this guide. + +## Practical steps + +### Step 1: Deploying Antrea + +```bash +kubectl apply -f https://github.com/vmware-tanzu/antrea/releases/download/v0.9.2/antrea.yml +``` + +You may also choose a different [Antrea +version](https://github.com/vmware-tanzu/antrea/releases). + +### Step 2: Deploy Multus as a DaemonSet + +```bash +git clone https://github.com/intel/multus-cni.git && cd multus-cni +cat ./images/multus-daemonset.yml | kubectl apply -f - +``` + +### Step 3: Create a NetworkAttachmentDefinition + +Make sure that you are using the correct interface name for `"master"`: it +should be the name of the Node's network interface (e.g. `eth0`) that you want +to use as the parent interface for the macvlan secondary Pod interfaces. If you +are using our [test cluster], `enp0s9` is the correct value. + +```bash +cat < +samplepod-7956c4498-9dz98 1/1 Running 0 68s 10.10.1.12 k8s-node-worker-1 +samplepod-7956c4498-ghrdg 1/1 Running 0 68s 10.10.1.13 k8s-node-worker-1 +samplepod-7956c4498-n65bn 1/1 Running 0 68s 10.10.2.12 k8s-node-worker-2 +samplepod-7956c4498-q6vp2 1/1 Running 0 68s 10.10.1.11 k8s-node-worker-1 +samplepod-7956c4498-xztf4 1/1 Running 0 68s 10.10.2.11 k8s-node-worker-2 +``` + +```bash +$ kubectl exec samplepod-7956c4498-65v6m -- ip addr +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +3: eth0@if18: mtu 1450 qdisc noqueue state UP group default + link/ether c2:ce:36:6b:ba:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.10.2.10/24 brd 10.10.2.255 scope global eth0 + valid_lft forever preferred_lft forever +4: net1@if4: mtu 1500 qdisc noqueue state UP group default + link/ether be:a0:35:f2:08:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.78.205/24 brd 192.168.78.255 scope global net1 + valid_lft forever preferred_lft forever +``` + +```bash +$ kubectl exec samplepod-7956c4498-9dz98 -- ip addr +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +3: eth0@if20: mtu 1450 qdisc noqueue state UP group default + link/ether 92:8f:8a:1d:a0:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.10.1.12/24 brd 10.10.1.255 scope global eth0 + valid_lft forever preferred_lft forever +4: net1@if4: mtu 1500 qdisc noqueue state UP group default + link/ether 22:6e:b1:0a:f3:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.78.202/24 brd 192.168.78.255 scope global net1 + valid_lft forever preferred_lft forever +``` + +```bash +$ kubectl exec samplepod-7956c4498-9dz98 -- ping -c 3 192.168.78.205 +PING 192.168.78.205 (192.168.78.205) 56(84) bytes of data. +64 bytes from 192.168.78.205: icmp_seq=1 ttl=64 time=0.846 ms +64 bytes from 192.168.78.205: icmp_seq=2 ttl=64 time=0.410 ms +64 bytes from 192.168.78.205: icmp_seq=3 ttl=64 time=0.507 ms + +--- 192.168.78.205 ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2013ms +rtt min/avg/max/mdev = 0.410/0.587/0.846/0.186 ms +``` + +### Overview of a test cluster Node + +The diagram below shows an overview of a K8s Node when using the [test cluster] +and following all the steps above. For the sake of completeness, we show the +DHCP server running on that Node, but as we use a Deployment with a single +replica, the server may be running on any worker Node in the cluster. + +Test cluster Node + +## Using [whereabouts] for IPAM + +If you do not already have a DHCP server for the underlying parent network and +you find that deploying one in-cluster is impractical, you may want to consider +using [whereabouts] to assign IP addresses to the secondary interfaces. When +using [whereabouts], follow steps 1 and 2 above, along with step 4 if you want +the Nodes to be able to communicate with the Pods using the secondary +network. + +The next step is to install the [whereabouts] plugin as follows: + +```bash +git clone https://github.com/dougbtv/whereabouts && cd whereabouts +kubectl apply -f ./doc/daemonset-install.yaml -f ./doc/whereabouts.cni.cncf.io_ippools.yaml +``` + +Then create a NetworkAttachmentDefinition like the one below, after ensuring +that `"master"` matches the name of the parent interface on the Nodes, and that +the `range` and `exclude` configuration parameters are correct for your cluster +(in particular, make sure that you exclude IP addresses assigned to Nodes). If +you are using our [test cluster], you can use the NetworkAttachmentDefinition +below as is. + +```bash +cat < + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile b/docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile new file mode 100644 index 00000000000..a87665c609a --- /dev/null +++ b/docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile @@ -0,0 +1,19 @@ +FROM ubuntu:20.04 as cni-binary + +LABEL maintainer="Antrea " +LABEL description="A Docker which runs the dhcp daemon from the containernetworking project." + +RUN apt-get update && \ + apt-get install -y --no-install-recommends wget ca-certificates + +# Leading dot is required for the tar command below +ENV CNI_PLUGINS="./dhcp" + +RUN mkdir -p /opt/cni/bin && \ + wget -q -O - https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz | tar xz -C /opt/cni/bin $CNI_PLUGINS + +FROM ubuntu:20.04 + +COPY --from=cni-binary /opt/cni/bin/* /usr/local/bin + +ENTRYPOINT ["dhcp", "daemon"] diff --git a/docs/cookbooks/multus/build/cni-dhcp-daemon/README.md b/docs/cookbooks/multus/build/cni-dhcp-daemon/README.md new file mode 100644 index 00000000000..88aa0fcc42f --- /dev/null +++ b/docs/cookbooks/multus/build/cni-dhcp-daemon/README.md @@ -0,0 +1,16 @@ +# cni-dhcp-daemon + +This Docker image can be used to run the [DHCP daemon from the +containernetworking +project](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp). + +If you need to build a new version of the image and push it to Dockerhub, you +can run the following: + +```bash +docker build -t antrea/cni-dhcp-daemon:latest . +docker push antrea/cni-dhcp-daemon:latest +``` + +The `docker push` command will fail if you do not have permission to push to the +`antrea` Dockerhub repository. diff --git a/docs/cookbooks/multus/resources/dhcp-daemon.yml b/docs/cookbooks/multus/resources/dhcp-daemon.yml new file mode 100644 index 00000000000..7c503180ce6 --- /dev/null +++ b/docs/cookbooks/multus/resources/dhcp-daemon.yml @@ -0,0 +1,52 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + namespace: kube-system + name: dhcp-daemon + labels: + environment: antrea-multus-demo + app: dhcp-daemon +spec: + selector: + matchLabels: + environment: antrea-multus-demo + app: dhcp-daemon + template: + metadata: + labels: + environment: antrea-multus-demo + app: dhcp-daemon + spec: + hostPID: true + hostNetwork: true + initContainers: + - name: sock-cleanup + image: ubuntu:20.04 + command: ["rm", "-f", "/run/cni/dhcp.sock"] + volumeMounts: + - name: run + mountPath: /run + containers: + - name: dhcp-daemon + securityContext: + privileged: true + image: antrea/cni-dhcp-daemon:latest + imagePullPolicy: Always + args: [""] + volumeMounts: + - name: run + mountPath: /run + volumes: + - hostPath: + path: /run + name: run + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical + tolerations: + - key: CriticalAddonsOnly + operator: Exists + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists diff --git a/docs/cookbooks/multus/resources/dhcp-server.yml b/docs/cookbooks/multus/resources/dhcp-server.yml new file mode 100644 index 00000000000..ed8194140f4 --- /dev/null +++ b/docs/cookbooks/multus/resources/dhcp-server.yml @@ -0,0 +1,58 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: dhcp-server-conf + namespace: kube-system + labels: + environment: antrea-multus-demo +data: + dhcpd.conf: | + subnet 192.168.78.0 netmask 255.255.255.0 { + range 192.168.78.200 192.168.78.250; + } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + namespace: kube-system + name: dhcp-server + labels: + environment: antrea-multus-demo + app: dhcp-server +spec: + replicas: 1 + selector: + matchLabels: + environment: antrea-multus-demo + app: dhcp-server + template: + metadata: + labels: + environment: antrea-multus-demo + app: dhcp-server + spec: + hostPID: true + hostNetwork: true + initContainers: + - name: wait-for-interface + image: busybox + imagePullPolicy: IfNotPresent + command: ["sh"] + args: ["-c", "while ! ip link show mac0 | grep -q ',UP,'; do echo 'Waiting for mac0'; sleep 1; done"] + containers: + - name: dhcp-server + image: networkboot/dhcpd:latest + imagePullPolicy: IfNotPresent + args: ["mac0"] + volumeMounts: + - mountPath: /data/dhcpd.conf + name: dhcp-server-conf + readOnly: true + subPath: dhcpd.conf + volumes: + - configMap: + name: dhcp-server-conf + name: dhcp-server-conf + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical diff --git a/docs/cookbooks/multus/resources/macvlan-host-init.yml b/docs/cookbooks/multus/resources/macvlan-host-init.yml new file mode 100644 index 00000000000..60605410894 --- /dev/null +++ b/docs/cookbooks/multus/resources/macvlan-host-init.yml @@ -0,0 +1,62 @@ +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: macvlan-host-init + namespace: kube-system + labels: + environment: antrea-multus-demo + app: macvlan-host-init +spec: + selector: + matchLabels: + environment: antrea-multus-demo + app: macvlan-host-init + template: + metadata: + labels: + environment: antrea-multus-demo + app: macvlan-host-init + spec: + hostNetwork: true + hostPID: true + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical + tolerations: + - key: CriticalAddonsOnly + operator: Exists + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists + containers: + - name: host-init + image: gcr.io/google-containers/startup-script:v1 + imagePullPolicy: IfNotPresent + securityContext: + privileged: true + env: + - name: STARTUP_SCRIPT + value: | + #! /bin/bash + + set -o errexit + set -o pipefail + set -o nounset + + if ip link show mac0; then + exit 0 + fi + + parent="enp0s9" + inet=$(ip addr show $parent | grep "inet " | awk '{ print $2 }') + # does not work with that version of iproute + # ip link dev $parent promisc on + ifconfig $parent promisc + ip addr flush dev $parent + + ip link add mac0 link $parent type macvlan mode bridge + ip addr add $inet dev mac0 + ip link set dev mac0 up + + echo "macvlan-host initialization completed" diff --git a/docs/cookbooks/multus/resources/netAttachDef.yml b/docs/cookbooks/multus/resources/netAttachDef.yml new file mode 100644 index 00000000000..650058a7530 --- /dev/null +++ b/docs/cookbooks/multus/resources/netAttachDef.yml @@ -0,0 +1,14 @@ +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + name: macvlan-conf +spec: + config: '{ + "cniVersion": "0.3.0", + "type": "macvlan", + "master": "enp0s9", + "mode": "bridge", + "ipam": { + "type": "dhcp" + } + }' diff --git a/docs/cookbooks/multus/resources/test.yml b/docs/cookbooks/multus/resources/test.yml new file mode 100644 index 00000000000..eb955f215f8 --- /dev/null +++ b/docs/cookbooks/multus/resources/test.yml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: samplepod + labels: + environment: antrea-multus-demo + app: samplepod +spec: + replicas: 6 + selector: + matchLabels: + environment: antrea-multus-demo + app: samplepod + template: + metadata: + labels: + environment: antrea-multus-demo + app: samplepod + annotations: + k8s.v1.cni.cncf.io/networks: macvlan-conf + spec: + containers: + - name: samplepod + command: ["/bin/bash", "-c", "PID=; trap 'kill $PID' TERM INT; sleep infinity & PID=$!; wait $PID"] + image: antoninbas/toolbox:latest + nodeSelector: + kubernetes.io/os: linux diff --git a/docs/cookbooks/multus/test/Vagrantfile b/docs/cookbooks/multus/test/Vagrantfile new file mode 100644 index 00000000000..2bd0cb387c2 --- /dev/null +++ b/docs/cookbooks/multus/test/Vagrantfile @@ -0,0 +1,64 @@ +VAGRANTFILE_API_VERSION = "2" + +NUM_WORKERS = 2 + +K8S_POD_NETWORK_CIDR = "10.10.0.0/16" + +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + config.vm.box = "ubuntu/bionic64" + + config.vm.provider "virtualbox" do |v| + v.memory = 2048 + # 2 CPUS required to initialize k8s cluster with "kubeadm init" + v.cpus = 2 + + v.customize [ + "modifyvm", :id, + "--nicpromisc3", "allow-all" + ] + end + + groups = { + "master" => ["k8s-node-master"], + "workers" => ["k8s-node-worker-[1:#{NUM_WORKERS}]"], + } + + config.vm.define "k8s-node-master" do |node| + node.vm.hostname = "k8s-node-master" + node_ip = "192.168.77.100" + node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.100", virtualbox__intnet: true + + node.vm.provision :ansible do |ansible| + ansible.playbook = "playbook/k8s.yml" + ansible.groups = groups + ansible.extra_vars = { + # Ubuntu bionic does not ship with python2 + ansible_python_interpreter:"/usr/bin/python3", + node_ip: node_ip, + node_name: "k8s-node-master", + k8s_pod_network_cidr: K8S_POD_NETWORK_CIDR, + k8s_api_server_ip: node_ip, + } + end + end + + (1..NUM_WORKERS).each do |node_id| + config.vm.define "k8s-node-worker-#{node_id}" do |node| + node.vm.hostname = "k8s-node-worker-#{node_id}" + node_ip = "192.168.77.#{100 + node_id}" + node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.#{100 + node_id}", virtualbox__intnet: true + + node.vm.provision :ansible do |ansible| + ansible.playbook = "playbook/k8s.yml" + ansible.groups = groups + ansible.extra_vars = { + ansible_python_interpreter:"/usr/bin/python3", + node_ip: node_ip, + node_name: "k8s-node-worker-#{node_id}", + } + end + end + end +end diff --git a/test/e2e/infra/vagrant/Vagrantfile b/test/e2e/infra/vagrant/Vagrantfile index 8b484901c1d..2bd0cb387c2 100644 --- a/test/e2e/infra/vagrant/Vagrantfile +++ b/test/e2e/infra/vagrant/Vagrantfile @@ -1,6 +1,6 @@ VAGRANTFILE_API_VERSION = "2" -NUM_WORKERS = 1 +NUM_WORKERS = 2 K8S_POD_NETWORK_CIDR = "10.10.0.0/16" @@ -11,6 +11,11 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| v.memory = 2048 # 2 CPUS required to initialize k8s cluster with "kubeadm init" v.cpus = 2 + + v.customize [ + "modifyvm", :id, + "--nicpromisc3", "allow-all" + ] end groups = { @@ -22,6 +27,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| node.vm.hostname = "k8s-node-master" node_ip = "192.168.77.100" node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.100", virtualbox__intnet: true node.vm.provision :ansible do |ansible| ansible.playbook = "playbook/k8s.yml" @@ -42,6 +48,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| node.vm.hostname = "k8s-node-worker-#{node_id}" node_ip = "192.168.77.#{100 + node_id}" node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.#{100 + node_id}", virtualbox__intnet: true node.vm.provision :ansible do |ansible| ansible.playbook = "playbook/k8s.yml"