Apply Pod Security Standards at the Cluster Level

Note

Pod Security admission (PSA) is enabled by default in v1.23 and later, as it has graduated to beta. Pod Security is an admission controller that carries out checks against the Kubernetes when new pods are created. This tutorial shows you how to enforce the Pod Security Standard at the cluster level which applies a standard configuration to all namespaces in a cluster.

To apply Pod Security Standards to specific namespaces, refer to Apply Pod Security Standards at the namespace level.

If you are running a version of Kubernetes other than v1.27, check the documentation for that version.

Install the following on your workstation:

This tutorial demonstrates what you can configure for a Kubernetes cluster that you fully control. If you are learning how to configure Pod Security Admission for a managed cluster where you are not able to configure the control plane, read Apply Pod Security Standards at the namespace level.

Pod Security Admission lets you apply built-in with the following modes: enforce, audit, and warn.

To gather information that helps you to choose the Pod Security Standards that are most appropriate for your configuration, do the following:

  1. Create a cluster with no Pod Security Standards applied:

    The output is similar to:

    1. Creating cluster "psa-wo-cluster-pss" ...
    2. ✓ Ensuring node image (kindest/node:v1.27.0) 🖼
    3. ✓ Preparing nodes 📦
    4. ✓ Writing configuration 📜
    5. ✓ Starting control-plane 🕹️
    6. ✓ Installing CNI 🔌
    7. ✓ Installing StorageClass 💾
    8. Set kubectl context to "kind-psa-wo-cluster-pss"
    9. You can now use your cluster with:
    11. kubectl cluster-info --context kind-psa-wo-cluster-pss
    13. Thanks for using kind! 😊

  2. Set the kubectl context to the new cluster:

    1. kubectl cluster-info --context kind-psa-wo-cluster-pss

    The output is similar to this:

    1. Kubernetes control plane is running at https://127.0.0.1:61350
    3. CoreDNS is running at https://127.0.0.1:61350/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
    5. To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

  3. Get a list of namespaces in the cluster:

    1. kubectl get ns
    1. NAME                 STATUS   AGE
    2. default              Active   9m30s
    3. kube-node-lease      Active   9m32s
    4. kube-public          Active   9m32s
    5. kube-system          Active   9m32s
    6. local-path-storage   Active   9m26s

  4. Use --dry-run=server to understand what happens when different Pod Security Standards are applied:

    1. Privileged

      1. kubectl label --dry-run=server --overwrite ns --all \
      2. pod-security.kubernetes.io/enforce=privileged

      The output is similar to:

    2. Baseline

      1. kubectl label --dry-run=server --overwrite ns --all \
      2. pod-security.kubernetes.io/enforce=baseline

      The output is similar to:

      1. namespace/default labeled
      2. namespace/kube-node-lease labeled
      3. namespace/kube-public labeled
      4. Warning: existing pods in namespace "kube-system" violate the new PodSecurity enforce level "baseline:latest"
      5. Warning: etcd-psa-wo-cluster-pss-control-plane (and 3 other pods): host namespaces, hostPath volumes
      6. Warning: kindnet-vzj42: non-default capabilities, host namespaces, hostPath volumes
      7. Warning: kube-proxy-m6hwf: host namespaces, hostPath volumes, privileged
      8. namespace/kube-system labeled
      9. namespace/local-path-storage labeled

From the previous output, you’ll notice that applying the privileged Pod Security Standard shows no warnings for any namespaces. However, baseline and restricted standards both have warnings, specifically in the kube-system namespace.

In this section, you apply the following Pod Security Standards to the latest version:

  • baseline standard in enforce mode.
  • restricted standard in warn and audit mode.

The baseline Pod Security Standard provides a convenient middle ground that allows keeping the exemption list short and prevents known privilege escalations.

Additionally, to prevent pods from failing in kube-system, you’ll exempt the namespace from having Pod Security Standards applied.

When you implement Pod Security Admission in your own environment, consider the following:

  2. Exempting the kube-system namespace allows pods to run as privileged in this namespace. For real world use, the Kubernetes project strongly recommends that you apply strict RBAC policies that limit access to kube-system, following the principle of least privilege. To implement the preceding standards, do the following:

  3. Create a configuration file that can be consumed by the Pod Security Admission Controller to implement these Pod Security Standards:

    1. mkdir -p /tmp/pss
    2. cat <<EOF > /tmp/pss/cluster-level-pss.yaml
    3. apiVersion: apiserver.config.k8s.io/v1
    4. kind: AdmissionConfiguration
    5. plugins:
    6. - name: PodSecurity
    7.   configuration:
    8.     apiVersion: pod-security.admission.config.k8s.io/v1
    9.     kind: PodSecurityConfiguration
    10.     defaults:
    11.       enforce: "baseline"
    12.       enforce-version: "latest"
    13.       audit: "restricted"
    14.       audit-version: "latest"
    15.       warn: "restricted"
    16.       warn-version: "latest"
    17.     exemptions:
    18.       usernames: []
    19.       runtimeClasses: []
    20.       namespaces: [kube-system]
    21. EOF

    Note: pod-security.admission.config.k8s.io/v1 configuration requires v1.25+. For v1.23 and v1.24, use . For v1.22, use v1alpha1.

  4. Configure the API server to consume this file during cluster creation:

    1. cat <<EOF > /tmp/pss/cluster-config.yaml
    2. kind: Cluster
    3. apiVersion: kind.x-k8s.io/v1alpha4
    4. nodes:
    6.   kubeadmConfigPatches:
    7.   - |
    8.     kind: ClusterConfiguration
    9.     apiServer:
    10.         extraArgs:
    11.           admission-control-config-file: /etc/config/cluster-level-pss.yaml
    12.         extraVolumes:
    13.             hostPath: /etc/config
    14.             mountPath: /etc/config
    15.             readOnly: false
    16.             pathType: "DirectoryOrCreate"
    17.   extraMounts:
    18.   - hostPath: /tmp/pss
    19.     containerPath: /etc/config
    20.     # optional: if set, the mount is read-only.
    21.     # default false
    22.     readOnly: false
    23.     # optional: if set, the mount needs SELinux relabeling.
    24.     # default false
    25.     selinuxRelabel: false
    26.     # optional: set propagation mode (None, HostToContainer or Bidirectional)
    27.     # see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
    28.     # default None
    29.     propagation: None
    30. EOF

    Note: If you use Docker Desktop with KinD on macOS, you can add /tmp as a Shared Directory under the menu item Preferences > Resources > File Sharing.

  5. Create a cluster that uses Pod Security Admission to apply these Pod Security Standards:

    The output is similar to this:

    1. Creating cluster "psa-with-cluster-pss" ...
    2.  ✓ Ensuring node image (kindest/node:v1.27.0) 🖼
    3.  ✓ Preparing nodes 📦
    4.  ✓ Writing configuration 📜
    5.  ✓ Starting control-plane 🕹️
    6.  ✓ Installing CNI 🔌
    7.  ✓ Installing StorageClass 💾
    8. Set kubectl context to "kind-psa-with-cluster-pss"
    9. You can now use your cluster with:
    11. kubectl cluster-info --context kind-psa-with-cluster-pss
    13. Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

  6. Point kubectl to the cluster:

    1. kubectl cluster-info --context kind-psa-with-cluster-pss

    The output is similar to this:

    1. Kubernetes control plane is running at https://127.0.0.1:63855
    2. CoreDNS is running at https://127.0.0.1:63855/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
    4. To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

  7. Create a Pod in the default namespace:

    1. kubectl apply -f https://k8s.io/examples/security/example-baseline-pod.yaml

    The pod is started normally, but the output includes a warning:

    1. Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
    2. pod/nginx created

Now delete the clusters which you created above by running the following command:

  1. kind delete cluster --name psa-with-cluster-pss
  • Run a to perform all the preceding steps at once:
    1. Create a Pod Security Standards based cluster level Configuration
    2. Create a file to let API server consume this configuration
    3. Create a cluster that creates an API server with this configuration
    4. Set kubectl context to this new cluster
    5. Create a minimal pod yaml file
    6. Apply this file to create a Pod in the new cluster
  • Pod Security Admission
  • Apply Pod Security Standards at the namespace level