This document describes steps to deploy Antrea to an AKS cluster or an AKS Engine cluster.
Antrea can be deployed to an AKS cluster either in networkPolicyOnly
mode or
in encap
mode.
In networkPolicyOnly
mode, Antrea enforces NetworkPolicies and implements
other services for the AKS cluster, while the Azure CNI takes care of Pod IPAM
and traffic routing across Nodes. For more information about networkPolicyOnly
mode, refer to this design document.
In encap
mode, Antrea is in charge of Pod IPAM and of all the networking
functions on the Nodes. Using encap
mode provides access to additional Antrea
features, such as Multicast, as inter-Node Pod traffic is encapsulated, and is
not handled directly by the Azure Virtual Network. Note that the caveats
which apply when deploying Antrea in encap
mode on EKS do not apply for AKS.
We recommend encap
mode, as it will give you access to the most Antrea
features.
Install the Azure Cloud CLI. Refer to Azure CLI installation guide
We recommend using the latest version available (use at least version 2.39.0).
You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI.
-
Create an AKS cluster
export RESOURCE_GROUP_NAME=aks-antrea-cluster export CLUSTER_NAME=aks-antrea-cluster export LOCATION=westus az group create --name $RESOURCE_GROUP_NAME --location $LOCATION az aks create \ --resource-group $RESOURCE_GROUP_NAME \ --name $CLUSTER_NAME \ --node-count 2 \ --network-plugin azure
Note Do not specify network-policy option.
-
Get AKS cluster credentials
az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME
-
Access your cluster
kubectl get nodes NAME STATUS ROLES AGE VERSION aks-nodepool1-84330359-vmss000000 Ready agent 6m21s v1.16.10 aks-nodepool1-84330359-vmss000001 Ready agent 6m25s v1.16.10
-
Prepare the cluster Nodes
Deploy
antrea-node-init
DaemonSet to enableazure cni
to operate in transparent mode.kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea-aks-node-init.yml
-
Deploy Antrea
To deploy a released version of Antrea, pick a deployment manifest from the list of releases. Note that AKS support was added in release 0.9.0, which means you cannot pick a release older than 0.9.0. For any given release
<TAG>
(e.g.v0.9.0
), you can deploy Antrea as follows:kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea-aks.yml
To deploy the latest version of Antrea (built from the main branch), use the checked-in deployment yaml:
kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea-aks.yml
The command will deploy a single replica of Antrea controller to the AKS cluster and deploy Antrea agent to every Node. After a successful deployment you should be able to see these Pods running in your cluster:
$ kubectl get pods --namespace kube-system -l app=antrea NAME READY STATUS RESTARTS AGE antrea-agent-bpj72 2/2 Running 0 40s antrea-agent-j2sjz 2/2 Running 0 40s antrea-controller-6f7468cbff-5sk4t 1/1 Running 0 43s antrea-node-init-6twqg 1/1 Running 0 2m antrea-node-init-mqsqr 1/1 Running 0 2m
-
Restart remaining Pods
Once Antrea is up and running, restart all Pods in all Namespaces (kube-system, etc) so they can be managed by Antrea.
kubectl delete pods -n kube-system $(kubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>' | awk '{ print $1 }') pod "coredns-544d979687-96xm9" deleted pod "coredns-544d979687-p7dfb" deleted pod "coredns-autoscaler-78959b4578-849k8" deleted pod "dashboard-metrics-scraper-5f44bbb8b5-5qkkx" deleted pod "kube-proxy-6qxdw" deleted pod "kube-proxy-h6d89" deleted pod "kubernetes-dashboard-785654f667-7twsm" deleted pod "metrics-server-85c57978c6-pwzcx" deleted pod "tunnelfront-649ff5fb55-5lxg7" deleted
AKS now officially supports Bring your own Container Network Interface (BYOCNI).
Thanks to this, you can deploy Antrea on AKS in encap
mode, and you will not
lose access to any functionality. Check the AKS BYOCNI documentation for
prerequisites, in particular for AKS version requirements.
You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI.
-
Create an AKS cluster
export RESOURCE_GROUP_NAME=aks-antrea-cluster export CLUSTER_NAME=aks-antrea-cluster export LOCATION=westus az group create --name $RESOURCE_GROUP_NAME --location $LOCATION az aks create \ --resource-group $RESOURCE_GROUP_NAME \ --name $CLUSTER_NAME \ --node-count 2 \ --network-plugin none
Notice
--network-plugin none
, which tells AKS not to install any CNI plugin. -
Get AKS cluster credentials
az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME
-
Access your cluster
kubectl get nodes NAME STATUS ROLES AGE VERSION aks-nodepool1-40948307-vmss000000 NotReady agent 18m v1.27.7 aks-nodepool1-40948307-vmss000001 NotReady agent 17m v1.27.7
The Nodes are supposed to report a
NotReady
Status, since no CNI plugin is installed yet.
You can use Helm to easily install Antrea (or any other supported installation method). Just make sure that you configure Antrea NodeIPAM:
# you may not need this:
helm repo add antrea https://charts.antrea.io
helm repo update
cat <<EOF >> values-aks.yml
nodeIPAM:
enable: true
clusterCIDRs: ["10.10.0.0/16"]
EOF
helm install -n kube-system -f values-aks.yml antrea antrea/antrea
For more information about how to configure Antrea Node IPAM, please refer to Antrea Node IPAM guide.
After a while, make sure that all your Nodes report a Ready
Status and that
all your Pods are running correctly. Some Pods, and in particular the
metrics-server
Pods, may restart once after installing Antrea; this is not an
issue.
After a successful installation, Pods should look like this:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system antrea-agent-bpskv 2/2 Running 0 7m34s
kube-system antrea-agent-pfqrn 2/2 Running 0 7m34s
kube-system antrea-controller-555b8c799d-wk8zz 1/1 Running 0 7m34s
kube-system cloud-node-manager-2nszz 1/1 Running 0 31m
kube-system cloud-node-manager-wj68q 1/1 Running 0 31m
kube-system coredns-789789675-2nwd7 1/1 Running 0 6m48s
kube-system coredns-789789675-lbkfn 1/1 Running 0 31m
kube-system coredns-autoscaler-649b947bbd-j5wqc 1/1 Running 0 31m
kube-system csi-azuredisk-node-4bnnl 3/3 Running 0 31m
kube-system csi-azuredisk-node-52nwd 3/3 Running 0 31m
kube-system csi-azurefile-node-2h66l 3/3 Running 0 31m
kube-system csi-azurefile-node-dhrf2 3/3 Running 0 31m
kube-system konnectivity-agent-5fc7989878-6nhwl 1/1 Running 0 31m
kube-system konnectivity-agent-5fc7989878-t2n6h 1/1 Running 0 30m
kube-system kube-proxy-96c9p 1/1 Running 0 31m
kube-system kube-proxy-x8g8s 1/1 Running 0 31m
kube-system metrics-server-5955767688-2hjvn 2/2 Running 0 3m45s
kube-system metrics-server-5955767688-vmcq7 2/2 Running 0 3m45s
Antrea is an integrated CNI of AKS Engine, and can be installed in
networkPolicyOnly
mode or encap
mode to an AKS Engine cluster as part of the
AKS Engine cluster deployment. To learn basics of AKS Engine cluster deployment,
please refer to AKS Engine Quickstart Guide.
To configure Antrea to enforce NetworkPolicies for the AKS Engine cluster,
"networkPolicy": "antrea"
needs to be set in kubernetesConfig
of the AKS
Engine cluster definition (Azure CNI will be used as the networkPlugin
):
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"kubernetesConfig": {
"networkPolicy": "antrea"
}
}
}
You can use the deployment template
examples/networkpolicy/kubernetes-antrea.json
to deploy an AKS Engine cluster with Antrea in networkPolicyOnly
mode:
$ aks-engine deploy --dns-prefix <dns-prefix> \
--resource-group <reource-group> \
--location westus2 \
--api-model examples/networkpolicy/kubernetes-antrea.json \
--auto-suffix
To deploy Antrea in encap
mode for an AKS Engine cluster, both
"networkPlugin": "antrea"
and "networkPolicy": "antrea"
need to be set in
kubernetesConfig
of the AKS Engine cluster definition:
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"kubernetesConfig": {
"networkPlugin": "antrea",
"networkPolicy": "antrea"
}
}
}
You can add "networkPlugin": "antrea"
to the deployment template
examples/networkpolicy/kubernetes-antrea.json
,
and use the template to deploy an AKS Engine cluster with Antrea in encap
mode.