Explore Termination Behavior for Pods And Their Endpoints

    There are often cases when you need to terminate a Pod - be it for upgrade or scale down. In order to improve application availability, it may be important to implement a proper active connections draining.

    This tutorial explains the flow of Pod termination in connection with the corresponding endpoint state and removal by using a simple nginx web server to demonstrate the concept.

    The following is the example of the flow described in the Termination of Pods document.

    Let’s say you have a Deployment containing of a single replica (just for demonstration purposes) and a Service:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: nginx-deployment
    5.   labels:
    6.     app: nginx
    7. spec:
    8.   replicas: 1
    9.   selector:
    10.     matchLabels:
    11.       app: nginx
    12.   template:
    13.     metadata:
    14.       labels:
    15.         app: nginx
    16.     spec:
    17.       terminationGracePeriodSeconds: 120 # extra long grace period
    18.       containers:
    19.       - name: nginx
    20.         image: nginx:latest
    21.         ports:
    22.         - containerPort: 80
    23.         lifecycle:
    24.           preStop:
    25.             exec:
    26.               # Real life termination may take any time up to terminationGracePeriodSeconds.
    27.               # In this example - just hang around for at least the duration of terminationGracePeriodSeconds,
    28.               # at 120 seconds container will be forcibly terminated.
    29.               # Note, all this time nginx will keep processing requests.
    30.               command: [
    31.                 "/bin/sh", "-c", "sleep 180"
    32.               ]
    35. apiVersion: v1
    36. kind: Service
    37. metadata:
    38.   name: nginx-service
    39. spec:
    40.   selector:
    41.     app: nginx
    42.   ports:
    43.     - protocol: TCP
    44.       port: 80
    45.       targetPort: 80
    1. kubectl get endpointslice

    The output is similar to this:

    You can see its status, and validate that there is one endpoint registered:

    1. kubectl get endpointslices -o json -l kubernetes.io/service-name=nginx-service

    The output is similar to this:

    1. {
    2.     "addressType": "IPv4",
    3.     "apiVersion": "discovery.k8s.io/v1",
    4.     "endpoints": [
    5.         {
    6.             "addresses": [
    7.                 "10.12.1.201"
    8.             ],
    9.             "conditions": {
    10.                 "ready": true,
    11.                 "serving": true,
    12.                 "terminating": false

    Now let’s terminate the Pod and validate that the Pod is being terminated respecting the graceful termination period configuration:

    All pods:

    1. kubectl get pods
    1. NAME                                READY   STATUS        RESTARTS      AGE
    2. nginx-deployment-7768647bf9-b4b9s   1/1     Terminating   0             4m1s
    3. nginx-deployment-7768647bf9-rkxlw   1/1     Running       0             8s

    You can see that the new pod got scheduled.

    While the new endpoint is being created for the new Pod, the old endpoint is still around in the terminating state:

    The output is similar to this:

    1. {
    2.     "addressType": "IPv4",
    3.     "apiVersion": "discovery.k8s.io/v1",
    4.     "endpoints": [
    5.         {
    7.             ],
    8.             "conditions": {
    9.                 "ready": false,
    10.                 "serving": true,
    11.                 "terminating": true
    12.             },
    13.             "nodeName": "gke-main-default-pool-dca1511c-d17b",
    14.             "targetRef": {
    15.                 "kind": "Pod",
    16.                 "name": "nginx-deployment-7768647bf9-b4b9s",
    17.                 "namespace": "default",
    18.                 "uid": "66fa831c-7eb2-407f-bd2c-f96dfe841478"
    19.             },
    20.             "zone": "us-central1-c"
    21.         },
    22.         {
    23.             "addresses": [
    24.                 "10.12.1.202"
    25.             ],
    26.             "conditions": {
    27.                 "ready": true,
    28.                 "serving": true,
    29.                 "terminating": false
    30.             },
    31.             "nodeName": "gke-main-default-pool-dca1511c-d17b",
    32.             "targetRef": {
    33.                 "kind": "Pod",
    34.                 "name": "nginx-deployment-7768647bf9-rkxlw",
    35.                 "namespace": "default",
    36.                 "uid": "722b1cbe-dcd7-4ed4-8928-4a4d0e2bbe35"
    37.             },
    38.             "zone": "us-central1-c"

    This allows applications to communicate their state during termination and clients (such as load balancers) to implement a connections draining functionality. These clients may detect terminating endpoints and implement a special logic for them.

    In Kubernetes, endpoints that are terminating always have their ready status set as as . This needs to happen for backward compatibility, so existing load balancers will not use it for regular traffic. If traffic draining on terminating pod is needed, the actual readiness can be checked as a condition serving.