Locality failover

    Before proceeding, be sure to complete the steps under before you begin.

    In this task, you will use the Sleep pod in region1.zone1 as the source of requests to the HelloWorld service. You will then trigger failures that will cause failover between localities in the following sequence:

    Locality failover sequence

    Internally, Envoy priorities are used to control failover. These priorities will be assigned as follows for traffic originating from the Sleep pod (in region1 zone1):

    Apply a DestinationRule that configures the following:

    • Outlier detection for the HelloWorld service. This is required in order for failover to function properly. In particular, it configures the sidecar proxies to know when endpoints for a service are unhealthy, eventually triggering a failover to the next locality.

    • policy that forces each HTTP request to use a new connection. This task utilizes Envoy’s drain function to force a failover to the next locality. Once drained, Envoy will reject new connection requests. Since each request uses a new connection, this results in failover immediately following a drain. This configuration is used for demonstration purposes only.

    Verify traffic stays in region1.zone1

    Call the HelloWorld service from the Sleep pod:

    1.   "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
    2.   app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    3.   -- curl -sSL helloworld.sample:5000/hello
    4. Hello version: region1.zone1, instance: helloworld-region1.zone1-86f77cd7b-cpxhv

    Verify that the version in the response is region1.zone.

    Repeat this several times and verify that the response is always the same.

    Next, trigger a failover to region1.zone2. To do this, you for HelloWorld in region1.zone1:

    Call the HelloWorld service from the Sleep pod:

    1. $ kubectl exec --context="${CTX_R1_Z1}" -n sample -c sleep \
    2.   "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
    3.   app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    4.   -- curl -sSL helloworld.sample:5000/hello
    5. Hello version: region1.zone2, instance: helloworld-region1.zone2-86f77cd7b-cpxhv

    The first call will fail, which triggers the failover. Repeat the command several more times and verify that the version in the response is always region1.zone2.

    Failover to region2.zone3

    Call the HelloWorld service from the Sleep pod:

    1.   "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
    2.   app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    3.   -- curl -sSL helloworld.sample:5000/hello
    4. Hello version: region2.zone3, instance: helloworld-region2.zone3-86f77cd7b-cpxhv

    The first call will fail, which triggers the failover. Repeat the command several more times and verify that the version in the response is always region2.zone3.

    Now trigger a failover to region3.zone4. As you did previously, configure the HelloWorld in region2.zone3 to fail when called:

    Call the HelloWorld service from the Sleep pod:

    1. $ kubectl exec --context="${CTX_R1_Z1}" -n sample -c sleep \
    2.   "$(kubectl get pod --context="${CTX_R1_Z1}" -n sample -l \
    3.   app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    4.   -- curl -sSL helloworld.sample:5000/hello
    5. Hello version: region3.zone4, instance: helloworld-region3.zone4-86f77cd7b-cpxhv

    The first call will fail, which triggers the failover. Repeat the command several more times and verify that the version in the response is always region3.zone4.

    Congratulations! You successfully configured locality failover!

    Next steps

    resources and files from this task.