For HPA to work correctly, service deployments should have resources request definitions for containers. Follow this hello-world example to test if HPA is working correctly.

    1. Copy the hello-world deployment manifest below.

      Hello World Manifest

    2. Deploy it to your cluster.

      1. # kubectl create -f <HELLO_WORLD_MANIFEST>

    3. Copy one of the HPAs below based on the metric type you’re using:

      Hello World HPA: Resource Metrics

      1. apiVersion: autoscaling/v2beta1
      2. kind: HorizontalPodAutoscaler
      3. metadata:
      4.   name: hello-world
      5.   namespace: default
      6. spec:
      7.   scaleTargetRef:
      8.     apiVersion: extensions/v1beta1
      9.     kind: Deployment
      10.     name: hello-world
      11.   minReplicas: 1
      12.   maxReplicas: 10
      13.   metrics:
      14.   - type: Resource
      15.     resource:
      16.       name: cpu
      17.       targetAverageUtilization: 50
      18.   - type: Resource
      19.     resource:
      20.       name: memory
      21.       targetAverageValue: 1000Mi

      Hello World HPA: Custom Metrics

      1. apiVersion: autoscaling/v2beta1
      2. kind: HorizontalPodAutoscaler
      3. metadata:
      4.   name: hello-world
      5.   namespace: default
      6. spec:
      7.   scaleTargetRef:
      8.     apiVersion: extensions/v1beta1
      9.     kind: Deployment
      10.     name: hello-world
      11.   minReplicas: 1
      12.   maxReplicas: 10
      13.   metrics:
      14.   - type: Resource
      15.     resource:
      16.       name: cpu
      17.       targetAverageUtilization: 50
      18.   - type: Resource
      19.     resource:
      20.       name: memory
      21.       targetAverageValue: 100Mi
      22.   - type: Pods
      23.     pods:
      24.       metricName: cpu_system
      25.       targetAverageValue: 20m

    4. View the HPA info and description. Confirm that metric data is shown.

      Resource Metrics

        1. # kubectl get hpa
        2. NAME          REFERENCE                TARGETS                     MINPODS   MAXPODS   REPLICAS   AGE
        3. hello-world   Deployment/hello-world   1253376 / 100Mi, 0% / 50%   1         10        1          6m
        4. # kubectl describe hpa
        5. Name:                                                  hello-world
        6. Namespace:                                             default
        7. Labels:                                                <none>
        8. Annotations:                                           <none>
        9. CreationTimestamp:                                     Mon, 23 Jul 2018 20:21:16 +0200
        10. Reference:                                             Deployment/hello-world
        11. Metrics:                                               ( current / target )
        12.   resource memory on pods:                             1253376 / 100Mi
        13.   resource cpu on pods  (as a percentage of request):  0% (0) / 50%
        14. Min replicas:                                          1
        15. Max replicas:                                          10
        16. Conditions:
        17.   Type            Status  Reason              Message
        18.   ----            ------  ------              -------
        19.   AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
        20.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        21.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        22. Events:           <none>

        Custom Metrics

      2. Enter the following command.

        1. # kubectl describe hpa

        You should receive the output that follows.

    5. Generate a load for the service to test that your pods autoscale as intended. You can use any load-testing tool (Hey, Gatling, etc.), but we’re using .

    6. Test that pod autoscaling works as intended.

      To Test Autoscaling Using Resource Metrics:

      Upscale to 2 Pods: CPU Usage Up to Target

      Use your load testing tool to scale up to two pods based on CPU Usage.

      1. View your HPA.

        1. # kubectl describe hpa

        You should receive output similar to what follows.

        1. Name:                                                  hello-world
        2. Namespace:                                             default
        3. Labels:                                                <none>
        4. Annotations:                                           <none>
        5. CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
        6. Reference:                                             Deployment/hello-world
        7. Metrics:                                               ( current / target )
        8.   resource memory on pods:                             10928128 / 100Mi
        9.   resource cpu on pods  (as a percentage of request):  56% (280m) / 50%
        10. Min replicas:                                          1
        11. Max replicas:                                          10
        12. Conditions:
        13.   Type            Status  Reason              Message
        14.   ----            ------  ------              -------
        15.   AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
        16.   ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
        17.   ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        18. Events:
        19.   Type    Reason             Age   From                       Message
        20.   ----    ------             ----  ----                       -------
        21.   Normal  SuccessfulRescale  13s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
        1.   # kubectl get pods

        You should receive output similar to what follows:

        1.   NAME                                                     READY     STATUS    RESTARTS   AGE
        2.   hello-world-54764dfbf8-k8ph2                             1/1       Running   0          1m
        3.   hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

        Upscale to 3 pods: CPU Usage Up to Target

      Use your load testing tool to upscale to 3 pods based on CPU usage with horizontal-pod-autoscaler-upscale-delay set to 3 minutes.

      1. Enter the following command.

        1. # kubectl describe hpa

        You should receive output similar to what follows

      2. Enter the following command to confirm three pods are running.

        1. # kubectl get pods

        You should receive output similar to what follows.

        1.   NAME                                                     READY     STATUS    RESTARTS   AGE
        2.   hello-world-54764dfbf8-f46kh                             0/1       Running   0          1m
        3.   hello-world-54764dfbf8-k8ph2                             1/1       Running   0          5m
        4.   hello-world-54764dfbf8-q6l4v                             1/1       Running   0          3h

        Downscale to 1 Pod: All Metrics Below Target

      Use your load testing to scale down to 1 pod when all metrics are below target for horizontal-pod-autoscaler-downscale-delay (5 minutes by default).

      1. Enter the following command.

        1. # kubectl describe hpa
        1.   Name:                                                  hello-world
        2.   Labels:                                                <none>
        3.   Annotations:                                           <none>
        4.   CreationTimestamp:                                     Mon, 23 Jul 2018 22:22:04 +0200
        5.   Reference:                                             Deployment/hello-world
        6.   Metrics:                                               ( current / target )
        8.     resource cpu on pods  (as a percentage of request):  0% (0) / 50%
        9.   Min replicas:                                          1
        10.   Max replicas:                                          10
        11.   Conditions:
        12.     Type            Status  Reason              Message
        13.     ----            ------  ------              -------
        14.     AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
        15.     ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
        16.     ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
        17.   Events:
        18.     Type    Reason             Age   From                       Message
        19.     ----    ------             ----  ----                       -------
        20.     Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
        21.     Normal  SuccessfulRescale  6m    horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
        22.     Normal  SuccessfulRescale  1s    horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
    1. **To Test Autoscaling Using Custom Metrics:**
    3.  Upscale to 2 Pods: CPU Usage Up to Target
    5. Use your load testing tool to upscale two pods based on CPU usage.
    7. 1.  Enter the following command.
    9.     ```
    10.     # kubectl describe hpa
    11.     ```
    13.     You should receive output similar to what follows.
    15.     ```
    16.     Name:                                                  hello-world
    17.     Namespace:                                             default
    18.     Labels:                                                <none>
    19.     Annotations:                                           <none>
    20.     CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    21.     Reference:                                             Deployment/hello-world
    22.     Metrics:                                               ( current / target )
    23.       resource memory on pods:                             8159232 / 100Mi
    24.       "cpu_system" on pods:                                7m / 20m
    25.       resource cpu on pods  (as a percentage of request):  64% (321m) / 50%
    26.     Min replicas:                                          1
    27.     Max replicas:                                          10
    28.     Conditions:
    29.       Type            Status  Reason              Message
    30.       ----            ------  ------              -------
    31.       AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
    32.       ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    33.       ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    34.     Events:
    35.       Type    Reason             Age   From                       Message
    36.       ----    ------             ----  ----                       -------
    37.       Normal  SuccessfulRescale  16s   horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    38.     ```
    40. 2.  Enter the following command to confirm two pods are running.
    42.     ```
    43.     # kubectl get pods
    44.     ```
    46.     You should receive output similar to what follows.
    48.     ```
    49.         NAME                           READY     STATUS    RESTARTS   AGE
    50.         hello-world-54764dfbf8-5pfdr   1/1       Running   0          3s
    51.         hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    52.     ```
    54.  Upscale to 3 Pods: CPU Usage Up to Target
    56. Use your load testing tool to scale up to three pods when the cpu\_system usage limit is up to target.
    58. 1.  Enter the following command.
    60.     ```
    61.     # kubectl describe hpa
    62.     ```
    64.     You should receive output similar to what follows:
    66.     ```
    67.       Name:                                                  hello-world
    68.       Namespace:                                             default
    69.       Labels:                                                <none>
    70.       Annotations:                                           <none>
    71.       CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    72.       Reference:                                             Deployment/hello-world
    73.       Metrics:                                               ( current / target )
    74.         resource memory on pods:                             8374272 / 100Mi
    75.         "cpu_system" on pods:                                27m / 20m
    76.         resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
    77.       Min replicas:                                          1
    78.       Max replicas:                                          10
    79.       Conditions:
    80.         Type            Status  Reason              Message
    81.         ----            ------  ------              -------
    82.         AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
    83.         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    84.         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    85.       Events:
    86.         Type    Reason             Age   From                       Message
    87.         ----    ------             ----  ----                       -------
    88.         Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    89.         Normal  SuccessfulRescale  3s    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
    90.     ```
    92. 2.  Enter the following command to confirm three pods are running.
    94.     ```
    95.     # kubectl get pods
    96.     ```
    97.     You should receive output similar to what follows:
    99.     ```
    100.       # kubectl get pods
    101.       NAME                           READY     STATUS    RESTARTS   AGE
    102.       hello-world-54764dfbf8-5pfdr   1/1       Running   0          3m
    103.       hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
    104.       hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    105.     ```
    107.  Upscale to 4 Pods: CPU Usage Up to Target
    109. Use your load testing tool to upscale to four pods based on CPU usage. `horizontal-pod-autoscaler-upscale-delay` is set to three minutes by default.
    111. 1.  Enter the following command.
    113.     ```
    114.     # kubectl describe hpa
    115.     ```
    117.     You should receive output similar to what follows.
    119.     ```
    120.       Name:                                                  hello-world
    121.       Namespace:                                             default
    122.       Labels:                                                <none>
    123.       Annotations:                                           <none>
    124.       CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    125.       Reference:                                             Deployment/hello-world
    126.       Metrics:                                               ( current / target )
    127.         resource memory on pods:                             8374272 / 100Mi
    128.         "cpu_system" on pods:                                27m / 20m
    129.         resource cpu on pods  (as a percentage of request):  71% (357m) / 50%
    130.       Min replicas:                                          1
    131.       Max replicas:                                          10
    132.       Conditions:
    133.         Type            Status  Reason              Message
    134.         ----            ------  ------              -------
    135.         AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 3
    136.         ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    137.         ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    138.       Events:
    139.         Type    Reason             Age   From                       Message
    140.         ----    ------             ----  ----                       -------
    141.         Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    142.         Normal  SuccessfulRescale  3m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
    143.         Normal  SuccessfulRescale  4s    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
    144.     ```
    146. 2.  Enter the following command to confirm four pods are running.
    148.     ```
    149.     # kubectl get pods
    150.     ```
    152.     You should receive output similar to what follows.
    154.     ```
    155.       NAME                           READY     STATUS    RESTARTS   AGE
    156.       hello-world-54764dfbf8-2p9xb   1/1       Running   0          5m
    157.       hello-world-54764dfbf8-5pfdr   1/1       Running   0          2m
    158.       hello-world-54764dfbf8-m2hrl   1/1       Running   0          1s
    159.       hello-world-54764dfbf8-q6l82   1/1       Running   0          6h
    160.     ```
    162.  Downscale to 1 Pod: All Metrics Below Target
    164. Use your load testing tool to scale down to one pod when all metrics below target for `horizontal-pod-autoscaler-downscale-delay`.
    166. 1.  Enter the following command.
    168.     ```
    169.     # kubectl describe hpa
    170.     ```
    172.     You should receive similar output to what follows.
    174.     ```
    175.         Name:                                                  hello-world
    176.         Namespace:                                             default
    177.         Labels:                                                <none>
    178.         Annotations:                                           <none>
    179.         CreationTimestamp:                                     Tue, 24 Jul 2018 18:01:11 +0200
    180.         Reference:                                             Deployment/hello-world
    181.         Metrics:                                               ( current / target )
    182.           resource memory on pods:                             8101888 / 100Mi
    183.           "cpu_system" on pods:                                8m / 20m
    184.           resource cpu on pods  (as a percentage of request):  0% (0) / 50%
    185.         Min replicas:                                          1
    186.         Max replicas:                                          10
    187.         Conditions:
    188.           Type            Status  Reason              Message
    189.           ----            ------  ------              -------
    190.           AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 1
    191.           ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
    192.           ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
    193.         Events:
    194.           Type    Reason             Age   From                       Message
    195.           ----    ------             ----  ----                       -------
    196.           Normal  SuccessfulRescale  10m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
    197.           Normal  SuccessfulRescale  8m    horizontal-pod-autoscaler  New size: 3; reason: pods metric cpu_system above target
    198.           Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
    199.           Normal   SuccessfulRescale             13s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
    200.     ```
    202. 2.  Enter the following command to confirm a single pods is running.
    204.     ```
    205.         # kubectl get pods
    206.     ```
    208.     You should receive output similar to what follows.
    210.     ```
    211.         NAME                           READY     STATUS    RESTARTS   AGE
    213.     ```