Quickstart

    Calico for Windows is a hybrid implementation that requires a Linux cluster for Calico components and Linux workloads, and Windows nodes for Windows workloads.

    Review the and the Calico for Windows requirements.

    Before beginning the quickstart, setup a Calico cluster on Linux nodes and provision Windows machines.

    For Linux control nodes using Calico networking, strict affinity must be set to . This is required to prevent Linux nodes from borrowing IP addresses from Windows nodes:

    note

    If the above command failed to find ipamconfigurations resource, you need to install Calico API server. Please refer to .

    The following steps install a Kubernetes cluster on a single Windows node with a Linux control node.

    • Kubernetes VXLAN

      The geeky details of what you get by default:

    • Kubernetes BGP

      The geeky details of what you get by default:

    • EKS

      The geeky details of what you get by default:

    • AKS

      The geeky details of what you get by default:

    • Kubernetes VXLAN

    • Kubernetes BGP
    • EKS
    • AKS

    1. Ensure that BGP is disabled since you’re using VXLAN. If you installed Calico using operator, you can do this by:

      1. kubectl patch installation default --type=merge -p '{"spec": {"calicoNetwork": {"bgp": "Disabled"}}}'

      If you installed Calico using the manifest then BGP is already disabled.

    2. Prepare the directory for Kubernetes files on Windows node.

      1. mkdir c:\k

    3. Copy the Kubernetes kubeconfig file from the master node (default, Location $HOME/.kube/config), to c:\k\config.

    4. Download the PowerShell script, install-calico-windows.ps1.

      1. Invoke-WebRequest https://github.com/projectcalico/calico/releases/download/v3.24.5/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1

    5. Install Calico for Windows for your datastore with using the default parameters or . The PowerShell script downloads Calico for Windows release binary, Kubernetes binaries, Windows utilities files, configures Calico for Windows, and starts the Calico service.

      Kubernetes datastore (default)

      1. c:\install-calico-windows.ps1 -KubeVersion <your Kubernetes version (e.g. 1.18.6)> `
      2.                               -ServiceCidr <your service cidr (default 10.96.0.0/12)> `
      3.                               -DNSServerIPs <your DNS service IP (default 10.96.0.10)>

      etcd datastore

      1. c:\install-calico-windows.ps1 -KubeVersion <your Kubernetes version (e.g. 1.18.6)> `
      2.                               -Datastore etcdv3 `
      3.                               -EtcdEndpoints <your etcd endpoint ip> `
      4.                               -EtcdTlsSecretName <your etcd TLS secret name in calico-system namespace> (default no etcd TLS secret is used) `
      5.                               -EtcdKey <path to key file> (default not using TLS) `
      6.                               -EtcdCert <path to cert file> (default not using TLS) `
      7.                               -EtcdCaCert <path to ca cert file> (default not using TLS) `
      8.                               -ServiceCidr <your service cidr (default 10.96.0.0/12)> `
      9.                               -DNSServerIPs <your DNS server IPs (default 10.96.0.10)>

      Quickstart - 图2note

      • You do not need to pass a parameter if the default value of the parameter is correct for your cluster.
      • If your Windows nodes have multiple network adapters, you can configure the one used for VXLAN by editing VXLAN_ADAPTER in C:\CalicoWindows\config.ps1, then restarting Calico for Windows.

    6. Verify that the Calico services are running.

      1. Get-Service -Name CalicoNode
      2. Get-Service -Name CalicoFelix

    7. Install and start kubelet/kube-proxy service. Execute following PowerShell script/commands.

      1. C:\CalicoWindows\kubernetes\install-kube-services.ps1
      2. Start-Service -Name kubelet
      3. Start-Service -Name kube-proxy

    8. Verify kubelet/kube-proxy services are running.

      1. Get-Service -Name kubelet
      2. Get-Service -Name kube-proxy

    9. Enable BGP service on Windows node (instead of VXLAN). Install the RemoteAccess service using the following Powershell commands:

      1. Install-WindowsFeature RemoteAccess
      2. Install-WindowsFeature RSAT-RemoteAccess-PowerShell
      3. Install-WindowsFeature Routing

      Then restart the computer:

      1. Restart-Computer -Force

      before running:

      1. Install-RemoteAccess -VpnType RoutingOnly

      Sometimes the remote access service fails to start automatically after install. To make sure it is running, execute the following command:

      1. Start-Service RemoteAccess

    10. Prepare the directory for Kubernetes files on Windows node.

      1. mkdir c:\k

    11. Copy the Kubernetes kubeconfig file from the master node (default, Location $HOME/.kube/config), to c:\k\config.

    12. Download the PowerShell script, install-calico-windows.ps1.

      1. Invoke-WebRequest https://github.com/projectcalico/calico/releases/download/v3.24.5/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1

    13. Install Calico for Windows for your datastore with using the default parameters or customize installation parameters. The PowerShell script downloads Calico for Windows release binary, Kubernetes binaries, Windows utilities files, configures Calico for Windows, and starts the Calico service.

      You do not need to pass a parameter if the default value of the parameter is correct for your cluster.

      Kubernetes datastore (default)

      1. c:\install-calico-windows.ps1 -KubeVersion <your Kubernetes version (e.g. 1.18.6)> `
      2.                               -ServiceCidr <your service cidr (default 10.96.0.0/12)> `
      3.                               -DNSServerIPs <your DNS service IP (default 10.96.0.10)>

      etcd datastore

      1. c:\install-calico-windows.ps1 -KubeVersion <your Kubernetes version (e.g. 1.18.6)> `
      2.                               -Datastore etcdv3 `
      3.                               -EtcdEndpoints <your etcd endpoint ip> `
      4.                               -EtcdTlsSecretName <your etcd TLS secret name in calico-system namespace> (default no etcd TLS secret is used) `
      5.                               -EtcdKey <path to key file> (default not using TLS) `
      6.                               -EtcdCert <path to cert file> (default not using TLS) `
      7.                               -EtcdCaCert <path to ca cert file> (default not using TLS) `
      8.                               -ServiceCidr <your service cidr (default 10.96.0.0/12)> `
      9.                               -DNSServerIPs <your DNS server IPs (default 10.96.0.10)>

      note

      You do not need to pass a parameter if the default value of the parameter is correct for your cluster.

      1. Get-Service -Name CalicoFelix

    15. Install and start kubelet/kube-proxy service. Execute following PowerShell script/commands.

      1. C:\CalicoWindows\kubernetes\install-kube-services.ps1
      2. Start-Service -Name kubelet
      3. Start-Service -Name kube-proxy

    16. Verify kubelet/kube-proxy services are running.

      1. Get-Service -Name kubelet
      2. Get-Service -Name kube-proxy

    17. Ensure that a Windows instance role has permissions to get namespaces and to get secrets in the calico-system namespace (or kube-system namespace if you are using a non operator-managed Calico installation.) One way to do this is by running the following comands to install the required permissions temporarily. Before running the commands, replace <eks_node_name> with the Kubernetes node name of the EKS Windows node, for example ip-192-168-42-34.us-west-2.compute.internal.

      Quickstart - 图4note

      If you are using a non operator-managed Calico installation, replace the namespace calico-system with kube-system in the commands below.

      1. kubectl create clusterrole calico-install-ns --verb=get --resource=namespace
      2. kubectl create clusterrolebinding calico-install-ns --clusterrole=calico-install-ns --user=system:node:<eks_node_name>
      3. kubectl create role calico-install-token --verb=get,list --resource=secrets --namespace calico-system
      4. kubectl create rolebinding calico-install-token --role=calico-install-token --user=system:node:<eks_node_name> --namespace calico-system

    18. Prepare the directory for Kubernetes files on the Windows node.

    19. and move the kubectl binary to c:\k.

    20. Download the PowerShell script, install-calico-windows.ps1.

      1. Invoke-WebRequest https://github.com/projectcalico/calico/releases/download/v3.24.5/install-calico-windows.ps1 -OutFile c:\install-calico-windows.ps1

    21. Install Calico for Windows for your datastore with using the default parameters or customize installation parameters. The PowerShell script downloads Calico for Windows release binary, Kubernetes binaries, Windows utilities files, configures Calico for Windows, and starts the Calico service.

      You do not need to pass a parameter if the default value of the parameter is correct for your cluster.

      Kubernetes datastore (default)

      1. c:\install-calico-windows.ps1 -ServiceCidr <your service cidr (default 10.96.0.0/12)> `
      2.                               -DNSServerIPs <your DNS service IP (default 10.96.0.10)>

      etcd datastore

      1. c:\install-calico-windows.ps1 -Datastore etcdv3 `
      2.                               -EtcdEndpoints <your etcd endpoint ip> `
      3.                               -ServiceCidr <your service cidr (default 10.96.0.0/12)> `
      4.                               -DNSServerIPs <your DNS server IPs (default 10.96.0.10)>

      note

      You do not need to pass a parameter if the default value of the parameter is correct for your cluster.

    22. Verify that the Calico services are running.

      1. Get-Service -Name CalicoNode
      2. Get-Service -Name CalicoFelix

    23. Verify kubelet and kube-proxy services are running.

      1. Get-Service -Name kubelet
      2. Get-Service -Name kube-proxy

    24. If you installed temporary RBAC in the first step, remove the permissions by running the following commands.

      Quickstart - 图6note

      If you are using a non operator-managed Calico installation, replace the namespace calico-system with in the commands below.

      1. kubectl delete clusterrolebinding calico-install-ns
      2. kubectl delete clusterrole calico-install-ns
      3. kubectl delete rolebinding calico-install-token --namespace calico-system
      4. kubectl delete role calico-install-token --namespace calico-system

    25. Register the EnableAKSWindowsCalico feature flag with the following Azure CLI command.

      1. az feature register --namespace "Microsoft.ContainerService" --name "EnableAKSWindowsCalico"

    26. Wait until the EnableAKSWindowsCalico feature flag is registered successfully. Execute following CLI command to get current status of the feature.

      1. az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/EnableAKSWindowsCalico')].{Name:name,State:properties.state}"

      Move to next step if the output from above command matches the following output.

      1. Name                                               State
      2. -------------------------------------------------  ----------
      3. Microsoft.ContainerService/EnableAKSWindowsCalico  Registered

    27. Refresh the registration of the Microsoft.ContainerService resource provider. Run the following command.

      1. az provider register --namespace Microsoft.ContainerService

    28. Create the AKS cluster with these settings: network-plugin to azure, and network-policy to calico. For example,

      1. az group create -n $your-resource-group -l $your-region
      2. az aks create \
      3.  --resource-group $your-resource-group \
      4.  --name $your-cluster-name \
      5.  --node-count 1 \
      6.  --enable-addons monitoring \
      7.  --windows-admin-username azureuser \
      8.  --windows-admin-password $your-windows-password \
      9.  --kubernetes-version 1.20.2 \
      10.  --vm-set-type VirtualMachineScaleSets \
      11.  --service-principal $your-service-principal \
      12.  --client-secret $your-client-secret \
      13.  --load-balancer-sku standard \
      14.  --node-vm-size Standard_D2s_v3 \
      15.  --network-plugin azure \
      16.  --network-policy calico

    29. Add a Windows node pool. For example,

      1. az aks nodepool add \
      2.  --resource-group $your-resource-group \
      3.  --cluster-name $your-cluster-name \
      4.  --os-type Windows \
      5.  --name $your-windows-node-pool-name \
      6.  --node-count 1 \
      7.  --kubernetes-version 1.20.2 \
      8.  --node-vm-size Standard_D2s_v3

    Congratulations! You now have a Kubernetes cluster with Calico for Windows and a Linux control node.

    note

    This installation method is a tech preview and should not be used for production clusters. Upgrades from a tech preview version of this installation method to the GA version might not be seamless.

    With Kubernetes v1.22, a new Windows container type called “HostProcess containers” can run directly on the host with access to the host network namespace, storage, and devices. With this feature, Calico for Windows can now be installed and managed using Kubernetes resources such as Daemonsets and ConfigMaps, instead of needing to configure and install Calico for Windows manually on each node. Using this installation method, the Calico for Windows services are no longer registered on the host. Instead, the services are run directly within HostProcess containers.

    Requirements

    In addition to the , this installation method has additional requirements:

    • Kubernetes v1.22+
    • HostProcess containers support enabled: for v1.22, HostProcess containers support has to be . For Kubernetes v1.23+, HostProcess containers are enabled by default.
    • ContainerD 1.6.0+
    • The Windows nodes have joined the cluster

    To install ContainerD on the Windows node and configure the ContainerD service:

    1. Invoke-WebRequest https://docs.tigera.io/calico/3.24/scripts/Install-Containerd.ps1 -OutFile c:\Install-Containerd.ps1
    2. c:\Install-Containerd.ps1 -ContainerDVersion 1.6.2 -CNIConfigPath "c:/etc/cni/net.d" -CNIBinPath "c:/opt/cni/bin"

    If you have an existing Calico for Windows installation using the manual method, your Windows nodes may have already joined the cluster.

    To join a Windows node to a cluster provisioned with kubeadm:

    • Install kubeadm and kubelet binaries and install the kubelet service
    1. Invoke-WebRequest https://docs.tigera.io/calico/3.24/scripts/PrepareNode.ps1 -OutFile c:\PrepareNode.ps1
    2. c:\PrepareNode.ps1 -KubernetesVersion v1.23.4 -ContainerRuntime ContainerD
    • Run kubeadm on a control plane host and copy the join command
    1. kubeadm token create --print-join-command
    • Edit the join command by appending --cri-socket "npipe:////./pipe/containerd-containerd" and update the kubeadm.exe path to c:\k\kubeadm.exe. An example join command:
      • Run the join command on the Windows node. Shortly after it completes successfully, the Windows node will appear in kubectl get nodes. The new node’s status will be NotReady since the Calico CNI has not yet been installed.

      Migrating from Calico for Windows installed manually

      If your Windows nodes already have Calico for Windows installed using the manual installation method, you can continue this quickstart guide to migrate to a manifest-based installation. This installation process will uninstall any existing Calico for Windows services and overwrite the Calico for Windows installation files with those included in the calico/windows image. If kubelet and kube-proxy were installed using C:\CalicoWindows\kubernetes\install-kube-services.ps1, those services will updated in-place and remain installed. If those services were running, they are restarted so those services will be updated in place and remain installed.

      Quickstart - 图8note

      Before proceeding, take note of the configuration parameters in C:\CalicoWindows\config.ps1. These configuration parameters will be needed during the install.

      Install

      • Kubernetes VXLAN
      • Kubernetes BGP

      1. Ensure that BGP is disabled.

        • If you installed Calico using the manifest, BGP is already disabled.
        • If you installed Calico using the operator, run this command:
        1. kubectl patch installation default --type=merge -p '{"spec": {"calicoNetwork": {"bgp": "Disabled"}}}'

      2. Download the Calico for Windows installation manifest.

        1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/calico-windows-vxlan.yaml -o calico-windows.yaml

      3. Get the cluster’s Kubernetes API server host and port, which will be used to update the Calico for Windows config map. The API server host and port is required so that the Calico for Windows installation script can create a kubeconfig file for Calico services. If your Windows nodes already have Calico for Windows installed manually, skip this step. The installation script will use the API server host and port from your node’s existing kubeconfig file if the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables are not provided in the calico-windows-config ConfigMap.

        First, make a note of the address of the API server:

        • If you have a single API server with a static IP address, you can use its IP address and port. The IP can be found by running:

          1. kubectl get endpoints kubernetes -o wide
          1. NAME         ENDPOINTS             AGE
          2. kubernetes   172.16.101.157:6443   40m

          If there are multiple entries under “ENDPOINTS”, then your cluster must have more than one API server. In this case, use the appropriate load balancing option below for your cluster.

        • If using DNS load balancing (as used by kops), use the FQDN and port of the API server api.internal.<clustername>.

        • If you have multiple API servers with a load balancer in front, you should use the IP and port of the load balancer.

        tip

        If your cluster uses a ConfigMap to configure kube-proxy you can find the “right” way to reach the API server by examining the config map. For example:

        In this case, the server is d881b853aea312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com and the port is 443 (the standard HTTPS port).

      4. Edit the calico-windows-config ConfigMap in the downloaded manifest and ensure the required variables are correct for your cluster.

        • CALICO_NETWORKING_BACKEND: This should be set to vxlan.
        • KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT: The Kubernetes API server host and port (discovered in the previous step) used to create a kubeconfig file for Calico services. If your node already has an existing kubeconfig file, leave these variables blank.
        • K8S_SERVICE_CIDR: The Kubernetes service clusterIP range configured in your cluster. This must match the service-cluster-ip-range used by kube-apiserver.
        • CNI_BIN_DIR: Path where Calico CNI binaries will be installed. This must match the CNI bin value in the ContainerD service configuration. If you used the provided Install-Containerd.ps1 script, you should use the CNI bin path value you provided to that script.
        • : Path where Calico CNI configuration will be installed. This must match the CNI conf value in the ContainerD service configuration. If you used the provided Install-Containerd.ps1 script, you should use the CNI conf path value you provided to that script.
        • DNS_NAME_SERVERS: The DNS nameservers that will be used in the CNI configuration.
        • FELIX_HEALTHENABLED: The Felix health check server must be enabled.

      5. Apply the Calico for Windows installation manifest.

        1. kubectl create -f calico-windows.yaml

      6. Monitor the installation.

        1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c install

        After the log Calico for Windows installed appears, installation is complete. Next, the Calico for Windows services are started in separate containers:

        1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c node
        2. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c felix

      7. Install kube-proxy.

        Depending on your platform, you may already have kube-proxy running on your Windows nodes. If kube-proxy is already running on your Windows nodes, skip this step. If kube-proxy is not running, you must install and run kube-proxy on each of the Windows nodes in your cluster. Note: The provided manifest depends on the kubeconfig provided by the kube-proxy ConfigMap in the kube-system namespace.

        • Download the kube-proxy manifest:

          1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/windows-kube-proxy.yaml -o windows-kube-proxy.yaml

        • Edit the downloaded manifest

          • Replace VERSION with your Windows nodes’ server version. E.g. 1809.
          • Update the K8S_VERSION env variable value with your Kubernetes cluster version.

        • Apply the manifest

          1. kubectl apply -f windows-kube-proxy.yaml

        • Verify the kube-proxy-windows daemonset is running

          1. kubectl describe ds -n kube-system kube-proxy-windows

      8. Enable BGP service on the Windows nodes. Install the RemoteAccess service using the following Powershell commands:

        1. Install-WindowsFeature RemoteAccess
        2. Install-WindowsFeature RSAT-RemoteAccess-PowerShell
        3. Install-WindowsFeature Routing

        Then restart the computer:

        1. Restart-Computer -Force

        before running:

        1. Install-RemoteAccess -VpnType RoutingOnly

        Sometimes the remote access service fails to start automatically after install. To make sure it is running, run the following command:

        1. Start-Service RemoteAccess

      9. Download the Calico for Windows installation manifest.

        1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/calico-windows-bgp.yaml -o calico-windows.yaml

      10. Get the cluster’s Kubernetes API server host and port, which will be used to update the Calico for Windows config map. The API server host and port is required so that the Calico for Windows installation script can create a kubeconfig file for Calico services. If your Windows nodes already have Calico for Windows installed manually, skip this step. The installation script will use the API server host and port from your node’s existing kubeconfig file if the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT variables are not provided in the calico-windows-config ConfigMap.

        First, make a note of the address of the API server:

        • If you have a single API server with a static IP address, you can use its IP address and port. The IP can be found by running:

          1. kubectl get endpoints kubernetes -o wide

          The output should look like the following, with a single IP address and port under “ENDPOINTS”:

          1. NAME         ENDPOINTS             AGE
          2. kubernetes   172.16.101.157:6443   40m

          If there are multiple entries under “ENDPOINTS”, then your cluster must have more than one API server. In this case, use the appropriate load balancing option below for your cluster.

        • If using DNS load balancing (as used by kops), use the FQDN and port of the API server api.internal.<clustername>.

        • If you have multiple API servers with a load balancer in front, you should use the IP and port of the load balancer.

        Quickstart - 图10tip

        If your cluster uses a ConfigMap to configure kube-proxy you can find the “right” way to reach the API server by examining the config map. For example:

        1. kubectl get configmap -n kube-system kube-proxy -o yaml | grep server`
        2. server: https://d881b853ae312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com

        In this case, the server is d881b853aea312e00302a84f1e346a77.gr7.us-west-2.eks.amazonaws.com and the port is 443 (the standard HTTPS port).

      11. Edit the calico-windows-config ConfigMap in the downloaded manifest and ensure the required variables are correct for your cluster.

        • CALICO_NETWORKING_BACKEND: This should be set to windows-bgp.
        • KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT: The Kubernetes API server host and port (discovered in the previous step) used to create a kubeconfig file for Calico services. If your node already has an existing kubeconfig file, leave these variables blank.
        • K8S_SERVICE_CIDR: The Kubernetes service clusterIP range configured in your cluster. This must match the service-cluster-ip-range used by kube-apiserver.
        • CNI_BIN_DIR: Path where Calico CNI binaries will be installed. This must match the CNI bin value in the ContainerD service configuration. If you used the provided Install-Containerd.ps1 script, you should use the CNI bin path value you provided to that script.
        • CNI_CONF_DIR: Path where Calico CNI configuration will be installed. This must match the CNI conf value in the ContainerD service configuration. If you used the provided Install-Containerd.ps1 script, you should use the CNI conf path value you provided to that script.
        • DNS_NAME_SERVERS: The DNS nameservers that will be used in the CNI configuration.
        • FELIX_HEALTHENABLED: The Felix health check server must be enabled.

      12. Apply the Calico for Windows installation manifest.

        1. kubectl create -f calico-windows.yaml

      13. Monitor the installation.

        1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c install

        After the log Calico for Windows installed appears, installation is complete. Next, the Calico for Windows services are started in separate containers:

        1. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c node
        2. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c felix
        3. kubectl logs -f -n calico-system -l k8s-app=calico-node-windows -c confd

      14. Install kube-proxy.

        Depending on your platform, you may already have kube-proxy running on your Windows nodes. If kube-proxy is already running on your Windows nodes, skip this step. If kube-proxy is not running, you must install and run kube-proxy on each of the Windows nodes in your cluster. Note: The provided manifest depends on the kubeconfig provided by the kube-proxy ConfigMap in the kube-system namespace.

        • Download the kube-proxy manifest:

          1. curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/windows-kube-proxy.yaml -o windows-kube-proxy.yaml

        • Edit the downloaded manifest

          • Replace VERSION with your Windows nodes’ server version. E.g. 1809.
          • Update the K8S_VERSION env variable value with your Kubernetes cluster version.

        • Apply the manifest

          1. kubectl apply -f windows-kube-proxy.yaml

        • Verify the kube-proxy-windows daemonset is running

        You can now use the Calico Linux-based docs site for your documentation. Before you continue, review the Limitations and known issues to understand the features (and sections of documentation) that do not apply to Windows.