- Overview
- Deploying Antrea on Windows worker Nodes
- Known issues
Antrea supports Windows worker Nodes. On Windows Nodes, Antrea sets up an overlay network to forward packets between Nodes and implements NetworkPolicies. Currently Geneve, VXLAN, and STT tunnels are supported.
This page shows how to install antrea-agent on Windows Nodes and register the Node to an existing Kubernetes cluster.
For the detailed design of how antrea-agent works on Windows, please refer to the design doc.
Note: Docker support on Windows Nodes was dropped completely in Antrea v2.0,
making containerd the only supported container runtime. As part of this
change, we renamed the antrea-windows-containerd.yml
manifest to
antrea-windows.yml
, and the antrea-windows-containerd-with-ovs.yml
manifest to antrea-windows-with-ovs.yml
. Prior to the Antrea v2.0 release,
the antrea-windows.yml
manifest was used to support Windows Nodes with
Docker. For the best experience, make sure that you refer to the version of
the documentation that matches the Antrea version you are deploying.
The following components should be configured and run on the Windows Node.
- kubernetes components
- OVS daemons
- antrea-agent
antrea-agent and the OVS daemons can either run as Pods (containerized) or as Windows services, and the following configurations are supported:
OVS daemons | antrea-agent | Supported | Refer to |
---|---|---|---|
Containerized | Containerized | Yes (recommended) | Installation as a Pod |
Containerized | Windows Service | No | N/A |
Windows Services | Containerized | Yes | Installation as a Pod |
Windows Services | Windows Services | Yes | Installation as a Service |
Watch this demo video of running Antrea in a Kubernetes cluster with both Linux and Windows Nodes. The demo also shows the Antrea OVS bridge configuration on a Windows Node, and NetworkPolicy enforcement for Windows Pods. Note, OVS driver and daemons are pre-installed on the Windows Nodes in the demo.
Running Antrea on Windows Nodes requires the containerd container runtime. The recommended installation method is Installation as a Pod, and it requires containerd 1.6 or higher. If you prefer running the Antrea Agent as a Windows service, or if you are using containerd 1.5, you can use the Installation as a Service method.
Starting from v2.1, Antrea Windows image is built on Linux host with docker buildx and uses hpc as the base image.
- Create a Kubernetes cluster.
- Obtain a Windows Server 2019 license (or higher) in order to configure the Windows Nodes that will host Windows containers. And install the latest Windows updates.
- On each Windows Node, install the following:
- Hyper-V with management tools. If your Nodes do not have the virtualization capabilities required by Hyper-V, use the workaround described in the Known issues section.
- containerd.
This installation method requires Antrea v1.10 or higher, and containerd 1.6 or higher (containerd 1.7 or higher is recommended). It relies on support for Windows HostProcess Pods, which is generally available starting with K8s 1.26.
More detailed containerd Version requirements are outlined below:
Kubernetes Version | Recommended containerd Version |
---|---|
1.26 | 1.7.0+, 1.6.18+ |
1.27 | 1.7.0+, 1.6.18+ |
1.28 | 1.7.0+, 1.6.18+ |
1.29 | 1.7.11+, 1.6.27+ |
1.30 | 1.7.13+, 1.6.28+ |
Note: Starting from Antrea v2.1, Antrea Windows image is built based on the HPC (Host Process Containers) image, containerd version 1.6.18 or higher is required because versions earlier than 1.6.18 do not support importing HPC images on Windows.
For more detailed information on Kubernetes-supported containerd versions, refer to the Containerd releases page
Starting with Antrea v1.13, Antrea takes over all the responsibilities of kube-proxy for Windows Nodes by default, and kube-proxy should not be deployed on Windows Nodes with Antrea.
Deploy Antrea for Linux on the control-plane Node following Getting started
document. The following command deploys Antrea with the version specified by <TAG>
:
kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea.yml
You need to manually set the kubeAPIServerOverride
field in the YAML
configuration file as the Antrea Proxy proxyAll
mode is enabled by default.
# Provide the address of Kubernetes apiserver, to override any value provided in kubeconfig or InClusterConfig.
# Defaults to "". It must be a host string, a host:port pair, or a URL to the base of the apiserver.
kubeAPIServerOverride: "10.10.1.1:6443"
# Option antreaProxy contains Antrea Proxy related configuration options.
antreaProxy:
# Option proxyAll tells antrea-agent to proxy ClusterIP Service traffic, regardless of where they come from.
# Therefore, running kube-proxy is no longer required. This requires the Antrea Proxy feature to be enabled.
# Note that this option is experimental. If kube-proxy is removed, option kubeAPIServerOverride must be used to access
# apiserver directly.
proxyAll: true
You can run both the Antrea Agent and the OVS daemons on Windows Nodes using a
single DaemonSet, by applying the file antrea-windows-with-ovs.yml
. This is
the recommended installation method. The following commands download the
manifest, set kubeAPIServerOverride
, and create the DaemonSet:
KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \
curl -sL https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea-windows-with-ovs.yml | \
sed "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \
kubectl apply -f -
Alternatively, to deploy the antrea-agent Windows DaemonSet without the OVS
daemons, apply the file antrea-windows.yml
with the following commands:
KUBE_APISERVER=$(kubectl config view -o jsonpath='{.clusters[0].cluster.server}') && \
curl -sL https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea-windows.yml | \
sed "s|.*kubeAPIServerOverride: \"\"| kubeAPIServerOverride: \"${KUBE_APISERVER}\"|g" | \
kubectl apply -f -
When using antrea-windows.yml
, you will need to install OVS
userspace daemons as services when you prepare your Windows worker Nodes, in the
next section.
Depending on which method you are using to install Antrea on Windows, and
depending on whether you are using your own signed
OVS kernel driver or you want to use the test-signed driver provided by Antrea,
you will need to invoke the Install-OVS.ps1
script differently (or not at all).
Containerized OVS daemons? | Test-signed OVS driver? | Run this command |
---|---|---|
Yes | Yes | Not required |
Yes | No | Not required |
No | Yes | .\Install-OVS.ps1 -InstallUserspace $true |
No | No | .\Install-OVS.ps1 -InstallUserspace $true -LocalFile <PathToOVSPackage> |
If you used antrea-windows-with-ovs.yml
to create the antrea-agent
Windows DaemonSet, then you are using "Containerized OVS daemons". For all other
methods, you are not using "Containerized OVS daemons".
Antrea provides a pre-built OVS package which contains a test-signed OVS kernel driver. If you don't have a self-signed OVS package and just want to try Antrea on Windows, this package can be used for testing.
[Test-only] If you are using test-signed driver (such as the one provided with Antrea), please make sure to enable test-signed:
Bcdedit.exe -set TESTSIGNING ON
Restart-Computer
If you want to run OVS as Windows native services, and you are bringing your own OVS package with a signed OVS kernel driver, you would run:
curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Install-OVS.ps1
.\Install-OVS.ps1 -InstallUserspace $true -LocalFile <PathToOVSPackage>
# verify that the OVS services are installed
get-service ovsdb-server
get-service ovs-vswitchd
Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled False
Firstly, install kubelet and kubeadm using the provided PrepareNode.ps1
script. Specify the Node IP, Kubernetes Version and container runtime while
running the script. The following command downloads and executes
Prepare-Node.ps1
:
# Example:
curl.exe -LO "https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Prepare-Node.ps1"
.\Prepare-Node.ps1 -KubernetesVersion v1.30.0 -NodeIP 192.168.1.10
Run the following commands to prepare the Node environment needed by antrea-agent:
mkdir c:\k\antrea
cd c:\k\antrea
$TAG="v2.0.0"
curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Clean-AntreaNetwork.ps1
curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Prepare-AntreaAgent.ps1
# use -RunOVSServices $false for containerized OVS!
.\Prepare-AntreaAgent.ps1 [-RunOVSServices $false]
The script Prepare-AntreaAgent.ps1
performs the following tasks:
-
Remove stale network resources created by antrea-agent.
After the Windows Node reboots, there will be stale network resources which need to be cleaned before starting antrea-agent.
-
Ensure OVS services are running.
This script starts OVS services on the Node if they are not running. This step needs to be skipped in case of OVS containerization. In that case, you need to specify the parameter
RunOVSServices
as false..\Prepare-AntreaAgent.ps1 -RunOVSServices $false
The script must be executed every time you restart the Node to prepare the environment for antrea-agent.
You can ensure that the script is executed automatically after each Windows startup by using different methods. Here are two examples for your reference:
- Example 1: Update kubelet service.
Insert following line in kubelet service script c:\k\StartKubelet.ps1
to invoke
Prepare-AntreaAgent.ps1
when starting kubelet service:
& C:\k\antrea\Prepare-AntreaAgent.ps1 -RunOVSServices $false
- Example 2: Create a ScheduledJob that runs at startup.
$trigger = New-JobTrigger -AtStartup -RandomDelay 00:00:30
$options = New-ScheduledJobOption -RunElevated
Register-ScheduledJob -Name PrepareAntreaAgent -Trigger $trigger -ScriptBlock { Invoke-Expression C:\k\antrea\Prepare-AntreaAgent.ps1 -RunOVSServices $false } -ScheduledJobOption $options
On Windows Nodes, run the kubeadm join
command to join the cluster. The token
is provided by the control-plane Node. If you lost the token, or the token has
expired, you can run kubeadm token create --print-join-command
(on the
control-plane Node) to generate a new token and join command. An example
kubeadm join
command is like below:
kubeadm join 192.168.101.5:6443 --token tdp0jt.rshv3uobkuoobb4v --discovery-token-ca-cert-hash sha256:84a163e57bf470f18565e44eaa2a657bed4da9748b441e9643ac856a274a30b9
There will be temporary network interruption on Windows worker Node on the first startup of antrea-agent. It's because antrea-agent will set the OVS to take over the host network. After that you should be able to view the Windows Nodes and Pods in your cluster by running:
# Show Nodes
kubectl get nodes -o wide -n kube-system
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
control-plane Ready control-plane 1h v1.29.0 10.176.27.168 <none> Ubuntu 22.04.3 LTS 6.2.0-1017-generic containerd://1.6.26
win-5akrf2tpq91 Ready <none> 1h v1.29.0 10.176.27.150 <none> Windows Server 2019 Datacenter 10.0.17763.5206 containerd://1.6.6
win-5akrf2tpq92 Ready <none> 1h v1.29.0 10.176.27.197 <none> Windows Server 2019 Datacenter 10.0.17763.5206 containerd://1.6.6
# Show antrea-agent Pods
kubectl get pods -o wide -n kube-system | grep windows
antrea-agent-windows-6hvkw 1/1 Running 0 100s
Install Antrea as usual. The following command deploys Antrea with the version
specified by <TAG>
:
kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea.yml
When running the Antrea Agent as a Windows service, no DaemonSet is created for
Windows worker Nodes. You will need to ensure that nssm is
installed on all your Windows Nodes. nssm
is a handy tool to manage services
on Windows.
To prepare your Windows worker Nodes, follow the steps in Join Windows worker Nodes.
With this installation method, OVS daemons are always run as services (not
containerized), and you will need to run Install-OVS.ps1
to install them.
When your Nodes are ready, run the following scripts to install the antrea-agent
service. NOTE: <KubernetesVersion>
, <KubeconfigPath>
and
<KubeletKubeconfigPath>
should be set by you. For example:
$KubernetesVersion="v1.29.0"
$KubeConfig="C:/Users/Administrator/.kube/config" # admin kubeconfig
$KubeletKubeconfigPath="C:/etc/kubernetes/kubelet.conf"
$TAG="v2.0.0"
$KubernetesVersion="<KubernetesVersion>"
$KubeConfig="<KubeconfigPath>"
$KubeletKubeconfigPath="<KubeletKubeconfigPath>"
$KubernetesHome="c:/k"
$AntreaHome="c:/k/antrea"
curl.exe -LO "https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Helper.psm1"
Import-Module ./Helper.psm1
Install-AntreaAgent -KubernetesVersion "$KubernetesVersion" -KubernetesHome "$KubernetesHome" -KubeConfig "$KubeConfig" -AntreaVersion "$TAG" -AntreaHome "$AntreaHome"
New-DirectoryIfNotExist "${AntreaHome}/logs"
nssm install antrea-agent "${AntreaHome}/bin/antrea-agent.exe" "--config=${AntreaHome}/etc/antrea-agent.conf --logtostderr=false --log_dir=${AntreaHome}/logs --alsologtostderr --log_file_max_size=100 --log_file_max_num=4"
nssm set antrea-agent DependOnService ovs-vswitchd
nssm set antrea-agent Start SERVICE_DELAYED_AUTO_START
Start-Service antrea-agent
Antrea also provides powershell scripts which help install and run the Antrea Agent manually, please complete the steps in Installation section, and skip the Add Windows antrea-agent DaemonSet step. Then run the following commands in powershell:
mkdir c:\k\antrea
cd c:\k\antrea
curl.exe -LO https://github.com/antrea-io/antrea/releases/download/<TAG>/Start-AntreaAgent.ps1
# Run antrea-agent
# $KubeConfigPath is the path of kubeconfig file
./Start-AntreaAgent.ps1 -kubeconfig $KubeConfigPath
Note: Some features such as supportbundle collection are not supported in this way. It's recommended to run antrea-agent as a Pod.
-
HNS Network is not persistent on Windows. So after the Windows Node reboots, the HNS Network created by antrea-agent is removed, and the Open vSwitch Extension is disabled by default. In this case, the stale OVS bridge and ports should be removed. A help script Clean-AntreaNetwork.ps1 can be used to clean the OVS bridge.
# If OVS userspace processes were running as a Service on Windows host ./Clean-AntreaNetwork.ps1 -OVSRunMode "service" # If OVS userspace processes were running inside container in antrea-agent Pod ./Clean-AntreaNetwork.ps1 -OVSRunMode "container"
-
Hyper-V feature cannot be installed on Windows Node due to the processor not having the required virtualization capabilities.
If the processor of the Windows Node does not have the required virtualization capabilities. The installation of Hyper-V feature will fail with the following error:
PS C:\Users\Administrator> Install-WindowsFeature Hyper-V Success Restart Needed Exit Code Feature Result ------- -------------- --------- -------------- False Maybe Failed {} Install-WindowsFeature : A prerequisite check for the Hyper-V feature failed. 1. Hyper-V cannot be installed: The processor does not have required virtualization capabilities. At line:1 char:1 + Install-WindowsFeature hyper-v + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (Hyper-V:ServerComponentWrapper) [Install-WindowsFeature], Exception + FullyQualifiedErrorId : Alteration_PrerequisiteCheck_Failed,Microsoft.Windows.ServerManager.Commands.AddWindowsF eatureCommand
The capabilities are required by the Hyper-V
hypervisor
components to support Hyper-V isolation. If you only need Process Isolation on the Nodes. You could apply the following workaround to skip CPU check for Hyper-V feature installation.# 1. Install containers feature Install-WindowsFeature containers # 2. Install Hyper-V management powershell module Install-WindowsFeature Hyper-V-Powershell # 3. Install Hyper-V feature without CPU check and disable the "hypervisor" dism /online /enable-feature /featurename:Microsoft-Hyper-V /all /NoRestart dism /online /disable-feature /featurename:Microsoft-Hyper-V-Online /NoRestart # 4. Restart-Computer to take effect Restart-Computer