Canary Upgrades

    When installing Istio, the installation setting can be used to deploy multiple independent control planes at the same time. A canary version of an upgrade can be started by installing the new Istio version’s control plane next to the old one, using a different revision setting. Each revision is a full Istio control plane implementation with its own Deployment, Service, etc.

    Before upgrading Istio, it is recommended to run the istioctl x precheck command to make sure the upgrade is compatible with your environment.

    When using revision-based upgrades jumping across two minor versions is supported (e.g. upgrading directly from version 1.8 to 1.10). This is in contrast to in-place upgrades where it is required to upgrade to each intermediate minor release.

    Control plane

    To install a new revision called canary, you would set the revision field as follows:

    In a production environment, a better revision name would correspond to the Istio version. However, you must replace . characters in the revision name, for example, revision=1-6-8 for Istio 1.6.8, because . is not a valid revision name character.

    1. $ istioctl install --set revision=canary

    After running the command, you will have two control plane deployments and services running side-by-side:

    1. $ kubectl get pods -n istio-system -l app=istiod
    2. NAME                                    READY   STATUS    RESTARTS   AGE
    3. istiod-786779888b-p9s5n                 1/1     Running   0          114m
    1. $ kubectl get svc -n istio-system -l app=istiod
    2. NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                                AGE
    3. istiod          ClusterIP   10.32.5.247   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                  33d
    4. istiod-canary   ClusterIP   10.32.6.58    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP,53/UDP,853/TCP   12m

    You will also see that there are two sidecar injector configurations including the new revision.

    1. $ kubectl get mutatingwebhookconfigurations
    2. NAME                            WEBHOOKS   AGE
    3. istio-sidecar-injector          1          7m56s
    4. istio-sidecar-injector-canary   1          3m18s

    Unlike istiod, Istio gateways do not run revision-specific instances, but are instead in-place upgraded to use the new control plane revision. You can verify that the istio-ingress gateway is using the revision by running the following command:

    However, simply installing the new revision has no impact on the existing sidecar proxies. To upgrade these, you must configure them to point to the new istiod-canary control plane. This is controlled during sidecar injection based on the namespace label istio.io/rev.

    To upgrade the namespace test-ns, remove the istio-injection label, and add the istio.io/rev label to point to the canary revision. The istio-injection label must be removed because it takes precedence over the istio.io/rev label for backward compatibility.

    1. $ kubectl label namespace test-ns istio-injection- istio.io/rev=canary

    After the namespace updates, you need to restart the pods to trigger re-injection. One way to do this is using:

    1. $ kubectl rollout restart deployment -n test-ns

    For example, the following command will show all the pods using the canary revision:

    1. $ kubectl get pods -n test-ns -l istio.io/rev=canary

    To verify that the new pods in the test-ns namespace are using the istiod-canary service corresponding to the canary revision, select one newly created pod and use the pod_name in the following command:

    1. $ istioctl proxy-status | grep ${pod_name} | awk '{print $7}'

    The output confirms that the pod is using istiod-canary revision of the control plane.

    Stable revision labels (experimental)

    If you’re using Helm, refer to the Helm upgrade documentation.

    Manually relabeling namespaces when moving them to a new revision can be tedious and error-prone. solve this problem. Revision tags are stable identifiers that point to revisions and can be used to avoid relabeling namespaces. Rather than relabeling the namespace, a mesh operator can simply change the tag to point to a new revision. All namespaces labeled with that tag will be updated at the same time.

    Consider a cluster with two revisions installed, 1-9-5 and 1-10-0. The cluster operator creates a revision tag prod-stable, pointed at the older, stable 1-9-5 version, and a revision tag prod-canary pointed at the newer 1-10-0 revision. That state could be reached via these commands:

    The resulting mapping between revisions, tags, and namespaces is as shown below:

    Two namespaces pointed to prod-stable and one pointed to prod-canary

    The cluster operator can view this mapping in addition to tagged namespaces through the command:

    1. $ istioctl tag list
    2. TAG         REVISION NAMESPACES
    3. prod-canary 1-10-0   ...
    4. prod-stable 1-9-5    ...

    After the cluster operator is satisfied with the stability of the control plane tagged with prod-canary, namespaces labeled istio.io/rev=prod-stable can be updated with one action by modifying the prod-stable revision tag to point to the newer 1-10-0 revision.

    1. $ istioctl tag set prod-stable --revision 1-10-0

    Namespace labels unchanged but now all namespaces pointed to 1-10-0

    Restarting injected workloads in the namespaces marked prod-stable will now result in those workloads using the 1-10-0 control plane. Notice that no namespace relabeling was required to migrate workloads to the new revision.

    Default tag

    The revision pointed to by the tag default is considered the default revision and has additional semantic meaning. The default revision performs the following functions:

    • Injects sidecars for the istio-injection=enabled namespace selector, the sidecar.istio.io/inject=true object selector, and the istio.io/rev=default selectors
    • Validates Istio resources
    • Steals the leader lock from non-default revisions and performs singleton mesh responsibilities (such as updating resource statuses)

    To make a revision 1-10-0 the default, run:

    1. $ istioctl tag set default --revision 1-10-0

    When using the default tag alongside an existing non-revisioned Istio installation it is recommended to remove the old MutatingWebhookConfiguration (typically called istio-sidecar-injector) to avoid having both the older and newer control planes attempt injection.

    After upgrading both the control plane and data plane, you can uninstall the old control plane. For example, the following command uninstalls a control plane of revision 1-6-5:

    1. $ istioctl uninstall --revision 1-6-5 -y

    If the old control plane does not have a revision label, uninstall it using its original installation options, for example:

    Confirm that the old control plane has been removed and only the new one still exists in the cluster:

    1. $ kubectl get pods -n istio-system -l app=istiod
    2. NAME                             READY   STATUS    RESTARTS   AGE

    Note that the above instructions only removed the resources for the specified control plane revision, but not cluster-scoped resources shared with other control planes. To uninstall Istio completely, refer to the uninstall guide.

    Uninstall canary control plane

    If you decide to rollback to the old control plane, instead of completing the canary upgrade, you can uninstall the canary revision using istioctl uninstall --revision=canary.

    Make sure to use the version corresponding to the old control plane to reinstall the old gateways and, to avoid downtime, make sure the old gateways are up and running before proceeding with the canary uninstall.