Managing graceful node shutdown

    During a graceful node shutdown, the kubelet sends a termination signal to pods running on the node and postpones the node shutdown until all the pods evicted. If a node unexpectedly shuts down, the graceful node shutdown feature minimizes interruption to workloads running on these pods.

    During a graceful node shutdown, the kubelet stops pods in two phases:

    • Regular pod termination

    • Critical pod termination

    You can define shutdown grace periods for regular and critical pods by configuring the following specifications in the custom resource:

    • shutdownGracePeriod: Specifies the total duration for pod termination for regular and critical pods.

    • shutdownGracePeriodCriticalPods: Specifies the duration for critical pod termination. This value must be less than the shutdownGracePeriod value.

    For example, if the shutdownGracePeriod value is 30s, and the shutdownGracePeriodCriticalPods value is 10s, the kubelet delays the node shutdown by 30 seconds. During the shutdown, the first 20 (30-10) seconds are reserved for gracefully shutting down regular pods, and the last 10 seconds are reserved for gracefully shutting down critical pods.

    To define a critical pod, assign a pod priority value greater than or equal to 2000000000. To define a regular pod, assign a pod priority value of less than 2000000000.

    Configuring graceful node shutdown

    To configure graceful node shutdown, create a KubeletConfig custom resource (CR) to specify a shutdown grace period for pods on a set of nodes. The graceful node shutdown feature minimizes interruption to workloads that run on these pods.

    Prerequisites

    • You have access to the cluster with the cluster-admin role.

    • You have defined priority classes for pods that require critical or regular classification.

    Procedure

    1. Define shutdown grace periods in the KubeletConfig CR by saving the following YAML in the kubelet-gns.yaml file:

    2. Create the KubeletConfig CR by running the following command:

      1. $ oc create -f kubelet-gns.yaml

      Example output

      1. kubeletconfig.machineconfiguration.openshift.io/graceful-shutdown created

    1. View the kubelet logs for a node to verify the grace period configuration by using the command line or by viewing the kubelet.conf file.

      1. To view the logs by using the command line, run the following command, replacing <node_name> with the name of the node:

        Example output

        1. Sep 12 22:13:46
        2. hyperkube[22317]: I0912 22:13:46.687472
        3. 22317 nodeshutdown_manager_linux.go:134]
        4. "Creating node shutdown manager"
        5. shutdownGracePeriodRequested="3m0s" (1)
        6. shutdownGracePeriodCriticalPods="2m0s"
        7. shutdownGracePeriodByPodPriority=[
        8. {Priority:0
        9. ShutdownGracePeriodSeconds:1200}
        10. {Priority:2000000000
        11. ShutdownGracePeriodSeconds:600}]
        12. ...

      2. To view the logs in the kubelet.conf file on a node, run the following commands to enter a debug session on the node:

        1. $ oc debug node/<node_name>

        Example output

        1. ...
        2.  “containerLogMaxSize”: “50Mi”,
        3.  “logging”: {
        4.   “flushFrequency”: 0,
        5.   “verbosity”: 0,
        6.   “options”: {
        7.    “json”: {
        8.     “infoBufferSize”: “0”
        9.    }
        10.   }
        11.  },
        12.  “shutdownGracePeriod”: “10m0s”, (1)
        13.  “shutdownGracePeriodCriticalPods”: “3m0s”
        14. }

    2. During a graceful node shutdown, you can verify that a pod was gracefully shut down by running the following command, replacing <pod_name> with the name of the pod:

      Example output

      1. Reason:         Terminated

    Additional resources