diff --git a/docs/cookbooks/multus/README.md b/docs/cookbooks/multus/README.md new file mode 100644 index 00000000000..480806690f5 --- /dev/null +++ b/docs/cookbooks/multus/README.md @@ -0,0 +1,326 @@ +# Using Antrea with Multus + +This guide will describe how to use Project Antrea with +[Multus](https://github.com/intel/multus-cni), in order to attach multiple +network interfaces to Pods. In this scenario, Antrea is used for the default +network, i.e. it is the CNI plugin which provisions the "primary" network +interface ("eth0") for each Pod. For the sake of this guide, we will use the +[macvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan) +CNI plugin to provision secondary network interfaces for selected Pods, but +similar steps should apply for other plugins as well, +e.g. [ipvlan](https://github.com/containernetworking/plugins/tree/master/plugins/main/ipvlan). + +## Prerequisites + +The only prerequisites are: + * a K8s cluster (Linux Nodes) running a K8s version supported by Antrea. At the + time of writing, we recommend version 1.16 or later. + * [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) + +All the required software will be deployed using YAML manifests, and the +corresponding container images will be downloaded from public registries. + +macvlan requires the network to be able to handle "promiscuous mode", as the +same physical interface / virtual adapter ends up being assigned multiple MAC +addresses. When using a virtual network for the Nodes, some configuration +changes are usually required, which depend on the virtualization technology. For +example: + * when using VirtualBox and [Internal + Networking](https://www.virtualbox.org/manual/ch06.html#network_internal), + set the `Promiscuous Mode` to `Allow All` + * when using VMware Fusion, enable "promiscuous mode" in the guest (Node) for + the appropriate interface (e.g. using `ifconfig`); this may prompt for your + password on the host unless you uncheck `Require authentication to enter + promiscuous mode` in the Network Preferences + +This needs to be done for every Node VM, so it's best if you can automate this +when provisioning your VMs. + +### Suggested test cluster + +If you need to create a K8s cluster to test this guide, we suggest you create +one by following [these +steps](https://github.com/vmware-tanzu/antrea/tree/master/test/e2e#creating-the-test-kubernetes-cluster-with-vagrant). You +will need to use a slightly modified Vagrantfile, which you can find +[here](test/Vagrantfile). Note that this Vagrantfile will create 3 VMs on your +machine, and each VM will be allocated 2GB of memory, so make sure you have +enough memory available. You can create the cluster with the following steps: + +```bash +git clone https://github.com/vmware-tanzu/antrea.git +cd antrea +cp docs/cookbooks/multus/test/Vagrantfile test/e2e/infra/vagrant/ +cd test/e2e/infra/vagrant +./infra/vagrant/provision.sh +``` + +The last command will take around 10 to 15 minutes to complete. After that, your +cluster is ready and you can set the `KUBECONFIG` environment variable in order +to use `kubectl`: + +```bash +export KUBECONFIG=`pwd`/infra/vagrant/playbook/kube/config +kubectl cluster-info +``` + +The cluster that you have created by following these steps is the one we will +use as an example in this guide. + +## Practical steps + +### Step 1: Deploying Antrea + +```bash +kubectl apply -f https://github.com/vmware-tanzu/antrea/releases/download/v0.9.2/antrea.yml +``` + +You may also choose a different [Antrea +version](https://github.com/vmware-tanzu/antrea/releases). + +### Step 2: Deploy Multus as a DaemonSet + +```bash +git clone https://github.com/intel/multus-cni.git && cd multus-cni +cat ./images/multus-daemonset.yml | kubectl apply -f - +``` + +### Step 3: Create a NetworkAttachmentDefinition + +Make sure that you are using the correct interface name for `"master"`: it +should be the name of the Node's network interface (e.g. `eth0`) that you want +to use as the parent interface for the macvlan secondary Pod interfaces. If you +are using our [test cluster], `enp0s9` is the correct value. + +```bash +cat < +samplepod-7956c4498-9dz98 1/1 Running 0 68s 10.10.1.12 k8s-node-worker-1 +samplepod-7956c4498-ghrdg 1/1 Running 0 68s 10.10.1.13 k8s-node-worker-1 +samplepod-7956c4498-n65bn 1/1 Running 0 68s 10.10.2.12 k8s-node-worker-2 +samplepod-7956c4498-q6vp2 1/1 Running 0 68s 10.10.1.11 k8s-node-worker-1 +samplepod-7956c4498-xztf4 1/1 Running 0 68s 10.10.2.11 k8s-node-worker-2 +``` + +```bash +$ kubectl exec samplepod-7956c4498-65v6m -- ip addr +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +3: eth0@if18: mtu 1450 qdisc noqueue state UP group default + link/ether c2:ce:36:6b:ba:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.10.2.10/24 brd 10.10.2.255 scope global eth0 + valid_lft forever preferred_lft forever +4: net1@if4: mtu 1500 qdisc noqueue state UP group default + link/ether be:a0:35:f2:08:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.78.205/24 brd 192.168.78.255 scope global net1 + valid_lft forever preferred_lft forever +``` + +```bash +$ kubectl exec samplepod-7956c4498-9dz98 -- ip addr +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +3: eth0@if20: mtu 1450 qdisc noqueue state UP group default + link/ether 92:8f:8a:1d:a0:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.10.1.12/24 brd 10.10.1.255 scope global eth0 + valid_lft forever preferred_lft forever +4: net1@if4: mtu 1500 qdisc noqueue state UP group default + link/ether 22:6e:b1:0a:f3:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.78.202/24 brd 192.168.78.255 scope global net1 + valid_lft forever preferred_lft forever +``` + +```bash +$ kubectl exec samplepod-7956c4498-9dz98 -- ping -c 3 192.168.78.205 +PING 192.168.78.205 (192.168.78.205) 56(84) bytes of data. +64 bytes from 192.168.78.205: icmp_seq=1 ttl=64 time=0.846 ms +64 bytes from 192.168.78.205: icmp_seq=2 ttl=64 time=0.410 ms +64 bytes from 192.168.78.205: icmp_seq=3 ttl=64 time=0.507 ms + +--- 192.168.78.205 ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2013ms +rtt min/avg/max/mdev = 0.410/0.587/0.846/0.186 ms +``` + +### Overview of a test cluster Node + +The diagram below shows an overview of a K8s Node when using the [test cluster] +and following all the steps above. For the sake of completeness, we show the +DHCP server running on that Node, but as we use a Deployment with a single +replica, the server may be running on any worker Node in the cluster. + +Test cluster Node + +## Using [whereabouts] for IPAM + +If you do not already have a DHCP server for the underlying parent network and +you find that deploying one in-cluster is impractical, you may want to consider +using [whereabouts] to assign IP addresses to the secondary interfaces. When +using [whereabouts], follow steps 1 and 2 above, along with step 4 if you want +the Nodes to be able to communicate with the Pods using the secondary +network. + +The next step is to install the [whereabouts] plugin as follows: + +```bash +git clone https://github.com/dougbtv/whereabouts && cd whereabouts +kubectl apply -f ./doc/daemonset-install.yaml -f ./doc/whereabouts.cni.cncf.io_ippools.yaml +``` + +Then create a NetworkAttachmentDefinition like the one below, after ensuring +that `"master"` matches the name of the parent interface on the Nodes, and that +the `range` and `exclude` configuration parameters are correct for your cluster +(in particular, make sure that you exclude IP addresses assigned to Nodes). If +you are using our [test cluster], you can use the NetworkAttachmentDefinition +below as is. + +```bash +cat < + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile b/docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile new file mode 100644 index 00000000000..a87665c609a --- /dev/null +++ b/docs/cookbooks/multus/build/cni-dhcp-daemon/Dockerfile @@ -0,0 +1,19 @@ +FROM ubuntu:20.04 as cni-binary + +LABEL maintainer="Antrea " +LABEL description="A Docker which runs the dhcp daemon from the containernetworking project." + +RUN apt-get update && \ + apt-get install -y --no-install-recommends wget ca-certificates + +# Leading dot is required for the tar command below +ENV CNI_PLUGINS="./dhcp" + +RUN mkdir -p /opt/cni/bin && \ + wget -q -O - https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz | tar xz -C /opt/cni/bin $CNI_PLUGINS + +FROM ubuntu:20.04 + +COPY --from=cni-binary /opt/cni/bin/* /usr/local/bin + +ENTRYPOINT ["dhcp", "daemon"] diff --git a/docs/cookbooks/multus/build/cni-dhcp-daemon/README.md b/docs/cookbooks/multus/build/cni-dhcp-daemon/README.md new file mode 100644 index 00000000000..88aa0fcc42f --- /dev/null +++ b/docs/cookbooks/multus/build/cni-dhcp-daemon/README.md @@ -0,0 +1,16 @@ +# cni-dhcp-daemon + +This Docker image can be used to run the [DHCP daemon from the +containernetworking +project](https://github.com/containernetworking/plugins/tree/master/plugins/ipam/dhcp). + +If you need to build a new version of the image and push it to Dockerhub, you +can run the following: + +```bash +docker build -t antrea/cni-dhcp-daemon:latest . +docker push antrea/cni-dhcp-daemon:latest +``` + +The `docker push` command will fail if you do not have permission to push to the +`antrea` Dockerhub repository. diff --git a/docs/cookbooks/multus/resources/dhcp-daemon.yml b/docs/cookbooks/multus/resources/dhcp-daemon.yml new file mode 100644 index 00000000000..7c503180ce6 --- /dev/null +++ b/docs/cookbooks/multus/resources/dhcp-daemon.yml @@ -0,0 +1,52 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + namespace: kube-system + name: dhcp-daemon + labels: + environment: antrea-multus-demo + app: dhcp-daemon +spec: + selector: + matchLabels: + environment: antrea-multus-demo + app: dhcp-daemon + template: + metadata: + labels: + environment: antrea-multus-demo + app: dhcp-daemon + spec: + hostPID: true + hostNetwork: true + initContainers: + - name: sock-cleanup + image: ubuntu:20.04 + command: ["rm", "-f", "/run/cni/dhcp.sock"] + volumeMounts: + - name: run + mountPath: /run + containers: + - name: dhcp-daemon + securityContext: + privileged: true + image: antrea/cni-dhcp-daemon:latest + imagePullPolicy: Always + args: [""] + volumeMounts: + - name: run + mountPath: /run + volumes: + - hostPath: + path: /run + name: run + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical + tolerations: + - key: CriticalAddonsOnly + operator: Exists + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists diff --git a/docs/cookbooks/multus/resources/dhcp-server.yml b/docs/cookbooks/multus/resources/dhcp-server.yml new file mode 100644 index 00000000000..ed8194140f4 --- /dev/null +++ b/docs/cookbooks/multus/resources/dhcp-server.yml @@ -0,0 +1,58 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: dhcp-server-conf + namespace: kube-system + labels: + environment: antrea-multus-demo +data: + dhcpd.conf: | + subnet 192.168.78.0 netmask 255.255.255.0 { + range 192.168.78.200 192.168.78.250; + } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + namespace: kube-system + name: dhcp-server + labels: + environment: antrea-multus-demo + app: dhcp-server +spec: + replicas: 1 + selector: + matchLabels: + environment: antrea-multus-demo + app: dhcp-server + template: + metadata: + labels: + environment: antrea-multus-demo + app: dhcp-server + spec: + hostPID: true + hostNetwork: true + initContainers: + - name: wait-for-interface + image: busybox + imagePullPolicy: IfNotPresent + command: ["sh"] + args: ["-c", "while ! ip link show mac0 | grep -q ',UP,'; do echo 'Waiting for mac0'; sleep 1; done"] + containers: + - name: dhcp-server + image: networkboot/dhcpd:latest + imagePullPolicy: IfNotPresent + args: ["mac0"] + volumeMounts: + - mountPath: /data/dhcpd.conf + name: dhcp-server-conf + readOnly: true + subPath: dhcpd.conf + volumes: + - configMap: + name: dhcp-server-conf + name: dhcp-server-conf + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical diff --git a/docs/cookbooks/multus/resources/macvlan-host-init.yml b/docs/cookbooks/multus/resources/macvlan-host-init.yml new file mode 100644 index 00000000000..60605410894 --- /dev/null +++ b/docs/cookbooks/multus/resources/macvlan-host-init.yml @@ -0,0 +1,62 @@ +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: macvlan-host-init + namespace: kube-system + labels: + environment: antrea-multus-demo + app: macvlan-host-init +spec: + selector: + matchLabels: + environment: antrea-multus-demo + app: macvlan-host-init + template: + metadata: + labels: + environment: antrea-multus-demo + app: macvlan-host-init + spec: + hostNetwork: true + hostPID: true + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-node-critical + tolerations: + - key: CriticalAddonsOnly + operator: Exists + - effect: NoSchedule + operator: Exists + - effect: NoExecute + operator: Exists + containers: + - name: host-init + image: gcr.io/google-containers/startup-script:v1 + imagePullPolicy: IfNotPresent + securityContext: + privileged: true + env: + - name: STARTUP_SCRIPT + value: | + #! /bin/bash + + set -o errexit + set -o pipefail + set -o nounset + + if ip link show mac0; then + exit 0 + fi + + parent="enp0s9" + inet=$(ip addr show $parent | grep "inet " | awk '{ print $2 }') + # does not work with that version of iproute + # ip link dev $parent promisc on + ifconfig $parent promisc + ip addr flush dev $parent + + ip link add mac0 link $parent type macvlan mode bridge + ip addr add $inet dev mac0 + ip link set dev mac0 up + + echo "macvlan-host initialization completed" diff --git a/docs/cookbooks/multus/resources/netAttachDef.yml b/docs/cookbooks/multus/resources/netAttachDef.yml new file mode 100644 index 00000000000..650058a7530 --- /dev/null +++ b/docs/cookbooks/multus/resources/netAttachDef.yml @@ -0,0 +1,14 @@ +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + name: macvlan-conf +spec: + config: '{ + "cniVersion": "0.3.0", + "type": "macvlan", + "master": "enp0s9", + "mode": "bridge", + "ipam": { + "type": "dhcp" + } + }' diff --git a/docs/cookbooks/multus/resources/test.yml b/docs/cookbooks/multus/resources/test.yml new file mode 100644 index 00000000000..eb955f215f8 --- /dev/null +++ b/docs/cookbooks/multus/resources/test.yml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: samplepod + labels: + environment: antrea-multus-demo + app: samplepod +spec: + replicas: 6 + selector: + matchLabels: + environment: antrea-multus-demo + app: samplepod + template: + metadata: + labels: + environment: antrea-multus-demo + app: samplepod + annotations: + k8s.v1.cni.cncf.io/networks: macvlan-conf + spec: + containers: + - name: samplepod + command: ["/bin/bash", "-c", "PID=; trap 'kill $PID' TERM INT; sleep infinity & PID=$!; wait $PID"] + image: antoninbas/toolbox:latest + nodeSelector: + kubernetes.io/os: linux diff --git a/docs/cookbooks/multus/test/Vagrantfile b/docs/cookbooks/multus/test/Vagrantfile new file mode 100644 index 00000000000..2bd0cb387c2 --- /dev/null +++ b/docs/cookbooks/multus/test/Vagrantfile @@ -0,0 +1,64 @@ +VAGRANTFILE_API_VERSION = "2" + +NUM_WORKERS = 2 + +K8S_POD_NETWORK_CIDR = "10.10.0.0/16" + +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + config.vm.box = "ubuntu/bionic64" + + config.vm.provider "virtualbox" do |v| + v.memory = 2048 + # 2 CPUS required to initialize k8s cluster with "kubeadm init" + v.cpus = 2 + + v.customize [ + "modifyvm", :id, + "--nicpromisc3", "allow-all" + ] + end + + groups = { + "master" => ["k8s-node-master"], + "workers" => ["k8s-node-worker-[1:#{NUM_WORKERS}]"], + } + + config.vm.define "k8s-node-master" do |node| + node.vm.hostname = "k8s-node-master" + node_ip = "192.168.77.100" + node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.100", virtualbox__intnet: true + + node.vm.provision :ansible do |ansible| + ansible.playbook = "playbook/k8s.yml" + ansible.groups = groups + ansible.extra_vars = { + # Ubuntu bionic does not ship with python2 + ansible_python_interpreter:"/usr/bin/python3", + node_ip: node_ip, + node_name: "k8s-node-master", + k8s_pod_network_cidr: K8S_POD_NETWORK_CIDR, + k8s_api_server_ip: node_ip, + } + end + end + + (1..NUM_WORKERS).each do |node_id| + config.vm.define "k8s-node-worker-#{node_id}" do |node| + node.vm.hostname = "k8s-node-worker-#{node_id}" + node_ip = "192.168.77.#{100 + node_id}" + node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.#{100 + node_id}", virtualbox__intnet: true + + node.vm.provision :ansible do |ansible| + ansible.playbook = "playbook/k8s.yml" + ansible.groups = groups + ansible.extra_vars = { + ansible_python_interpreter:"/usr/bin/python3", + node_ip: node_ip, + node_name: "k8s-node-worker-#{node_id}", + } + end + end + end +end diff --git a/test/e2e/infra/vagrant/Vagrantfile b/test/e2e/infra/vagrant/Vagrantfile index 8b484901c1d..2bd0cb387c2 100644 --- a/test/e2e/infra/vagrant/Vagrantfile +++ b/test/e2e/infra/vagrant/Vagrantfile @@ -1,6 +1,6 @@ VAGRANTFILE_API_VERSION = "2" -NUM_WORKERS = 1 +NUM_WORKERS = 2 K8S_POD_NETWORK_CIDR = "10.10.0.0/16" @@ -11,6 +11,11 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| v.memory = 2048 # 2 CPUS required to initialize k8s cluster with "kubeadm init" v.cpus = 2 + + v.customize [ + "modifyvm", :id, + "--nicpromisc3", "allow-all" + ] end groups = { @@ -22,6 +27,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| node.vm.hostname = "k8s-node-master" node_ip = "192.168.77.100" node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.100", virtualbox__intnet: true node.vm.provision :ansible do |ansible| ansible.playbook = "playbook/k8s.yml" @@ -42,6 +48,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| node.vm.hostname = "k8s-node-worker-#{node_id}" node_ip = "192.168.77.#{100 + node_id}" node.vm.network "private_network", ip: node_ip + node.vm.network "private_network", ip: "192.168.78.#{100 + node_id}", virtualbox__intnet: true node.vm.provision :ansible do |ansible| ansible.playbook = "playbook/k8s.yml"