Guide for scheduling Windows containers in Kubernetes

    • Configure an example deployment to run Windows containers on the Windows node
    • (Optional) Configure an Active Directory Identity for your Pod using Group Managed Service Accounts (GMSA)

    Before you begin

    • Create a Kubernetes cluster that includes a control plane and a
    • It is important to note that creating and deploying services and workloads on Kubernetes behaves in much the same way for Linux and Windows containers. Kubectl commands to interface with the cluster are identical. The example in the section below is provided to jumpstart your experience with Windows containers.

    To deploy a Windows container on Kubernetes, you must first create an example application. The example YAML file below creates a simple webserver application. Create a service spec named with the contents below:

    Note: Port mapping is also supported, but for simplicity in this example the container port 80 is exposed directly to the service.

    1. Check that all nodes are healthy:

      1. kubectl get nodes

    2. Deploy the service and watch for pod updates:

      When the service is deployed correctly both Pods are marked as Ready. To exit the watch command, press Ctrl+C.

    3. Check that the deployment succeeded. To verify:

      • Two containers per pod on the Windows node, use docker ps
      • Two pods listed from the Linux control plane node, use kubectl get pods
      • Node-to-pod communication across the network, curl port 80 of your pod IPs from the Linux control plane node to check for a web server response
      • Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) using docker exec or kubectl exec
      • Service-to-pod communication, curl the virtual service IP (seen under kubectl get services) from the Linux control plane node and from individual pods
      • Service discovery, curl the service name with the Kubernetes
      • Inbound connectivity, curl the NodePort from the Linux control plane node or machines outside of the cluster

    Note: Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. Only Windows pods are able to access service IPs.

    Observability

    Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files to all your containers and add the necessary entrypoints for LogMonitor to push your logs to STDOUT.

    Starting with Kubernetes v1.16, Windows containers can be configured to run their entrypoints and processes with different usernames than the image defaults. The way this is achieved is a bit different from the way it is done for Linux containers. Learn more about it .

    Managing Workload Identity with Group Managed Service Accounts

    Starting with Kubernetes v1.14, Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. Learn more about configuring and using GMSA for Windows containers .

    Users today need to use some combination of taints and node selectors in order to keep Linux and Windows workloads on their respective OS-specific nodes. This likely imposes a burden only on Windows users. The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads.

    Note:

    If the IdentifyPodOS feature gate is enabled, you can (and should) set .spec.os.name for a Pod to indicate the operating system that the containers in that Pod are designed for. For Pods that run Linux containers, set .spec.os.name to linux. For Pods that run Windows containers, set .spec.os.name to Windows.

    The scheduler does not use the value of .spec.os.name when assigning Pods to nodes. You should use normal Kubernetes mechanisms for to ensure that the control plane for your cluster places pods onto nodes that are running the appropriate operating system. no effect on the scheduling of the Windows pods, so taints and tolerations and node selectors are still required to ensure that the Windows pods land onto appropriate Windows nodes.

    • kubernetes.io/os = [windows|linux]
    • kubernetes.io/arch = [amd64|arm64|…]

    If a Pod specification does not specify a nodeSelector like "kubernetes.io/os": windows, it is possible the Pod can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector.

    However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. In those situations, you may be hesitant to make the configuration change to add nodeSelectors. The alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only.

    For example: --register-with-taints='os=windows:NoSchedule'

    By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector and the appropriate matching toleration to choose Windows.

    1. nodeSelector:
    2.     kubernetes.io/os: windows
    3.     node.kubernetes.io/windows-build: '10.0.17763'
    4. tolerations:
    5.     - key: "os"
    6.       operator: "Equal"
    8.       effect: "NoSchedule"

    The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows Server versions in the same cluster, then you should set additional node labels and nodeSelectors.

    Kubernetes 1.17 automatically adds a new label node.kubernetes.io/windows-build to simplify this. If you’re running an older version, then it’s recommended to add this label manually to Windows nodes.

    This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used today for each Windows Server version.

    1. Save this file to runtimeClasses.yml. It includes the appropriate nodeSelector for the Windows OS, architecture, and version.
    1. Run kubectl create -f runtimeClasses.yml using as a cluster administrator
    2. Add runtimeClassName: windows-2019 as appropriate to Pod specs

    For example:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: iis-2019
    5.   labels:
    6.     app: iis-2019
    7. spec:
    8.   replicas: 1
    9.   template:
    10.     metadata:
    11.       name: iis-2019
    12.       labels:
    13.         app: iis-2019
    14.       runtimeClassName: windows-2019
    15.       containers:
    16.       - name: iis
    18.         resources:
    19.           limits:
    20.             cpu: 1
    21.             memory: 800Mi
    22.           requests:
    23.             cpu: .1
    24.             memory: 300Mi
    25.         ports:
    26.           - containerPort: 80
    27.  selector:
    28.     matchLabels:
    29.       app: iis-2019
    30. ---
    31. apiVersion: v1
    32. kind: Service
    33. metadata:
    34.   name: iis
    35. spec:
    36.   type: LoadBalancer
    37.   ports:
    38.   - protocol: TCP
    39.     port: 80
    40.     app: iis-2019