Deploying Antrea on Windows

    This page shows how to install antrea-agent on Windows Nodes and register the Node to an existing Kubernetes cluster.

    For the detailed design of how antrea-agent works on Windows, please refer to the .

    The following components should be configured and run on the Windows Node.

    • OVS daemons
    • antrea-agent
    • kube-proxy

    antrea-agent and kube-proxy run as processes on host and are managed by management Pods. It is recommended to run OVS daemons as Windows services. If you don’t want to run antrea-agent and kube-proxy from the management Pods Antrea also provides scripts which help install and run these two components directly without Pod, please see Manually run kube-proxy and antrea-agent on Windows worker Nodes section for details.

    Antrea Windows demo

    Watch this demo video of running Antrea in a Kubernetes cluster with both Linux and Windows Nodes. The demo also shows the Antrea OVS bridge configuration on a Windows Node, NetworkPolicy enforcement for Windows Pods, and Antrea Traceflow from Octant. Note, OVS driver and daemons are pre-installed on the Windows Nodes in the demo.

    • Obtain a Windows Server 2019 license (or higher) in order to configure the Windows Node that hosts Windows containers. And install the latest Windows updates.
    • Deploy a Linux-based Kubernetes cluster.
    • Install with management tools. If your Nodes do not have the virtualization capabilities required by Hyper-V, you could try the workaround described in the Known issues section.
    • Install .
    • Install OVS and configure the daemons as Windows service.
      • The kernel driver of OVS should be .
      • If OVS driver is not signed, please refer to the Windows doc about how to install a test-signed driver package on the test computer.
      • If you don’t have a self-signed OVS package and just want to try the Antrea on Windows, Antrea provides a test-signed OVS package for you. See details in section.
    • Some manifests are from sig-windows-tool repo. Release version v0.1.5 has been verified.

    Installation as a Service (Containerd based runtimes)

    First install Antrea (v0.13.0+ is required for Containerd).

    Then, you can run the following commands. nssm will install Antrea as a Windows service. Please ensure is on your machine, which is a handy tool to manage services on Windows. NOTE: <KubernetesVersion>, <KubeconfigPath> <KubeProxyKubeconfigPath> and <KubeletKubeconfigPath> should be set by you. E.g.

    1. $KubernetesVersion="v1.23.5"
    2. $KubeConfig="C:/Users/Administrator/.kube/config" # admin kubeconfig
    3. $KubeletKubeconfigPath="C:/etc/kubernetes/kubelet.conf"
    4. $KubeProxyKubeconfigPath="C:/Users/Administrator/kubeproxy.conf"
    1. $TAG="v1.6.0"
    2. $KubernetesVersion="<KubernetesVersion>"
    3. $KubeConfig="<KubeconfigPath>"
    4. $KubeletKubeconfigPath="<KubeletKubeconfigPath>"
    5. $KubeProxyKubeconfigPath="<KubeProxyKubeconfigPath>"
    6. $KubernetesHome="c:/k"
    7. $AntreaHome="c:/k/antrea"
    8. $KubeProxyLogPath="c:/var/log/kube-proxy"
    10. curl.exe -LO "https://raw.githubusercontent.com/antrea-io/antrea/${TAG}/hack/windows/Helper.psm1"
    11. Import-Module ./Helper.psm1
    13. Install-AntreaAgent -KubernetesVersion "$KubernetesVersion" -KubernetesHome "$KubernetesHome" -KubeConfig "$KubeConfig" -AntreaVersion "$TAG" -AntreaHome "$AntreaHome"
    14. New-KubeProxyServiceInterface
    16. New-DirectoryIfNotExist "${AntreaHome}/logs"
    17. New-DirectoryIfNotExist "${KubeProxyLogPath}"
    18. nssm install antrea-agent "${AntreaHome}/bin/antrea-agent.exe" "--config=${AntreaHome}/etc/antrea-agent.conf --logtostderr=false --log_dir=${AntreaHome}/logs --alsologtostderr --log_file_max_size=100 --log_file_max_num=4"
    20. nssm set antrea-agent DependOnService kube-proxy ovs-vswitchd
    21. nssm set antrea-agent Start SERVICE_DELAYED_AUTO_START
    23. Start-Service kube-proxy
    24. Start-Service antrea-agent

    Installing Antrea using wins gives you a lot of flexibility to manage it as a Pod, but currently this only works with Docker due to a bug in the way Containerd handles host networking for Windows Pods ( ). In any case, if you are using Docker on Windows, this is how you can run Antrea in a Pod.

    Download & Configure Antrea for Linux

    Configure the Antrea for Linux on the control-plane Node following document.

    1. # Example:
    2. kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/antrea.yml

    Add Windows kube-proxy DaemonSet

    Add Windows-compatible versions of kube-proxy by applying file kube-proxy.yaml.

    Download kube-proxy.yaml from kubernetes official repository and set kube-proxy version.

    1. # Example:
    2. curl -L "https://github.com/kubernetes-sigs/sig-windows-tools/releases/download/v0.1.5/kube-proxy.yml" | sed 's/VERSION/v1.23.5/g' > kube-proxy.yml

    Replace the content of run-script.ps1 in configmap named kube-proxy-windows as following:

    1. apiVersion: v1
    2. data:
    3.   run-script.ps1: |-
    4.     $ErrorActionPreference = "Stop";
    5.     mkdir -force /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount
    6.     mkdir -force /host/k/kube-proxy
    9.     cp -force /var/lib/kube-proxy/* /host/var/lib/kube-proxy
    10.     cp -force /var/run/secrets/kubernetes.io/serviceaccount/* /host/var/lib/kube-proxy/var/run/secrets/kubernetes.io/serviceaccount
    12.     wins cli process run --path /k/kube-proxy/kube-proxy.exe --args "--v=3 --config=/var/lib/kube-proxy/config.conf --proxy-mode=userspace --hostname-override=$env:NODE_NAME"    
    14. kind: ConfigMap
    15. metadata:
    16.   labels:
    17.     app: kube-proxy
    18.   name: kube-proxy-windows
    19.   namespace: kube-system

    Set the hostNetwork option as true in spec of kube-proxy-windows daemonset.

    1. apiVersion: apps/v1
    2. kind: DaemonSet
    3. metadata:
    4.   labels:
    5.     k8s-app: kube-proxy
    6.   name: kube-proxy-windows
    7.   namespace: kube-system
    8. spec:
    9.   selector:
    10.     matchLabels:
    11.       k8s-app: kube-proxy-windows
    12.   template:
    13.     metadata:
    14.       labels:
    15.         k8s-app: kube-proxy-windows
    16.     spec:
    17.       hostNetwork: true
    1. kubectl apply -f kube-proxy.yml

    Add Windows antrea-agent DaemonSet

    Now you can deploy antrea-agent Windows DaemonSet by applying file antrea-windows.yml.

    Download and apply antrea-windows.yml.

    Join Windows worker Nodes

    1. (Optional) Install OVS (provided by Antrea or your own)

    Antrea provides a pre-built OVS package which contains test-signed OVS kernel driver. If you don’t have a self-signed OVS package and just want to try the Antrea on Windows, this package can be used for testing. We also provide a helper script Install-OVS.ps1 to install the OVS driver and register userspace binaries as services. If you want to use your own signed OVS package for production, you can run Install-OVS.ps1 like this:

    1. Install-OVS.ps1 -ImportCertificate $false -Local -LocalFile <PathToOVSPackage>

    [Test-only] First, if you are using test-signed driver (such as the one provided with Antrea), please make sure to enable test-signed

    1. Bcdedit.exe -set TESTSIGNING ON
    2. Restart-Computer

    Then, install the OVS using the script.

    1. curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Install-OVS.ps1
    2. .\Install-OVS.ps1 # Test-only

    Verify the OVS services are installed.

    1. get-service ovsdb-server
    2. get-service ovs-vswitchd

    2. Disable Windows Firewall

    1. Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled False

    3. Install wins, kubelet, kubeadm and configure kubelet startup params

    Firstly, install wins, kubelet, kubeadm using script PrepareNode.ps1 provided by kubernetes. The third component is used to run kube-proxy and antrea-agent on Windows host inside the Windows container.

    1. # Example:
    2. curl.exe -LO "https://github.com/kubernetes-sigs/sig-windows-tools/releases/download/v0.1.5/PrepareNode.ps1"
    3. .\PrepareNode.ps1 -KubernetesVersion v1.23.5

    4. Prepare Node environment needed by antrea-agent

    Run the following commands to prepare the Node environment needed by antrea-agent:

    2. cd c:\k\antrea
    3. curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Clean-AntreaNetwork.ps1
    4. curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Prepare-ServiceInterface.ps1
    5. curl.exe -LO https://raw.githubusercontent.com/antrea-io/antrea/main/hack/windows/Prepare-AntreaAgent.ps1
    6. .\Prepare-AntreaAgent.ps1

    The script Prepare-AntreaAgent.ps1 performs following tasks:

    • Prepare network adapter for kube-proxy.

      kube-proxy needs a network adapter to configure Kubernetes Services IPs and uses the adapter for proxying connections to Service. Use following script to create the network adapter. The adapter will be deleted automatically by Windows after the Windows Node reboots.

    • Remove stale network resources created by antrea-agent.

      After the Windows Node reboots, there will be stale network resources which need to be cleaned before starting antrea-agent.

    You could make the script be executed automatically after Windows startup by using different methods. Here’re two examples for your reference:

    • Example1: Update kubelet service.

    Insert following line in kubelet service script c:\k\StartKubelet.ps1 to invoke Prepare-AntreaAgent.ps1 when starting kubelet service:

    • Example2: Create a ScheduledJob that runs at startup.
    1. $trigger = New-JobTrigger -AtStartup -RandomDelay 00:00:30 
    2. $options = New-ScheduledJobOption -RunElevated
    3. Register-ScheduledJob -Name PrepareAntreaAgent -Trigger $trigger  -ScriptBlock { Invoke-Expression C:\k\antrea\Prepare-AntreaAgent.ps1 } -ScheduledJobOption $options

    5. Run kubeadm to join the Node

    On Windows Node, run the kubeadm join command to join the cluster. The token is provided by the control-plane Node.

    If you forgot the token, or the token has expired, you can run kubeadm token create --print-join-command (on the control-plane Node) to generate a new token and join command.

    1. # Example:
    2. kubeadm join 192.168.101.5:6443 --token tdp0jt.rshv3uobkuoobb4v  --discovery-token-ca-cert-hash sha256:84a163e57bf470f18565e44eaa2a657bed4da9748b441e9643ac856a274a30b9

    Then, set the Node IP used by kubelet. Open file /var/lib/kubelet/kubeadm-flags.env:

    1. KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:1.3.0"

    Append --node-ip=$NODE_IP at the end of params. Replace $NODE_IP with the address for kubelet. It should look like:

    1. KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=mcr.microsoft.com/oss/kubernetes/pause:1.3.0 --node-ip=$NODE_IP"

    Restart kubelet service for changes to take effect.

    1. restart-service kubelet

    Verify your installation

    There will be temporary network interruption on Windows worker Node on the first startup of antrea-agent. It’s because antrea-agent will set the OVS to take over the host network. After that you should be able to view the Windows Nodes and Pods in your cluster by running:

    1. # Show Nodes
    2. kubectl get nodes -o wide -n kube-system
    3. NAME                           STATUS   ROLES                  AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                  KERNEL-VERSION     CONTAINER-RUNTIME
    4. control-plane                  Ready    control-plane,master   1h    v1.23.5   10.176.27.168   <none>        Ubuntu 20.04.3 LTS                        5.11.0-1022-generic   docker://20.10.14
    5. win-5akrf2tpq91                Ready    <none>                 1h    v1.23.5   10.176.27.150   <none>        Windows Server 2019 Standard Evaluation   10.0.17763.2686       docker://20.10.9
    6. win-5akrf2tpq92                Ready    <none>                 1h    v1.23.5   10.176.27.197   <none>        Windows Server 2019 Standard Evaluation   10.0.17763.2686       docker://20.10.9
    8. # Show antrea-agent and kube-proxy Pods
    9. kubectl get pods -o wide -n kube-system | grep windows
    10. antrea-agent-windows-6hvkw                             1/1     Running     0          100s
    11. kube-proxy-windows-2d45w                               1/1     Running     0          102s

    Manually run kube-proxy and antrea-agent on Windows worker Nodes

    Aside from starting kube-proxy and antrea-agent from the management Pods, Antrea also provides powershell scripts which help install and run these two components directly without Pod. Please complete the steps in Installation section, skip and Add Windows antrea-agent DaemonSet steps. And then run the following commands in powershell.

    1. mkdir c:\k\antrea
    2. cd c:\k\antrea
    3. curl.exe -LO https://github.com/antrea-io/antrea/releases/download/<TAG>/Start-AntreaAgent.ps1
    4. # $KubeConfigPath is the path of kubeconfig file
    5. ./Start-AntreaAgent.ps1 -kubeconfig $KubeConfigPath

    1. HNS Network is not persistent on Windows. So after the Windows Node reboots, the HNS Network created by antrea-agent is removed, and the Open vSwitch Extension is disabled by default. In this case, the stale OVS bridge and ports should be removed. A help script Clean-AntreaNetwork.ps1 can be used to clean the OVS bridge.

    2. Hyper-V feature cannot be installed on Windows Node due to the processor not having the required virtualization capabilities.

      If the processor of the Windows Node does not have the required virtualization capabilities. The installation of Hyper-V feature will fail with the following error:

      1. # 1. Install containers feature
      2. Install-WindowsFeature containers
      4. # 2. Install Hyper-V management powershell module
      5. Install-WindowsFeature Hyper-V-Powershell
      7. # 3. Install Hyper-V feature without CPU check and disable the "hypervisor"
      8. dism /online /enable-feature /featurename:Microsoft-Hyper-V /all /NoRestart
      9. dism /online /disable-feature /featurename:Microsoft-Hyper-V-Online /NoRestart