Upgrading Kubernetes in Kubespray

    Description

    • docker_version
    • kube_version
    • etcd_version
    • calico_version
    • weave_version
    • flannel_version
    • kubedns_version

    Unsafe upgrade example

    If you wanted to upgrade just kube_version from v1.4.3 to v1.4.6, you coulddeploy the following way:

    And then repeat with v1.4.6 as kube_version:

    Graceful upgrade

    Kubespray also supports cordon, drain and uncordoning of nodes when performinga cluster upgrade. There is a separate playbook used for this purpose. It isimportant to note that upgrade-cluster.yml can only be used for upgrading anexisting cluster. That means there must be at least 1 kube-master alreadydeployed.

    1. git fetch origin
    2. git checkout origin/master
    3. ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.6.0

    After a successul upgrade, the Server Version should be updated:

    1. $ kubectl version
    3. Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0+coreos.0", GitCommit:"8031716957d697332f9234ddf85febb07ac6c3e3", GitTreeState:"clean", BuildDate:"2017-03-29T04:33:09Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

    Upgrade order

    • etcd
    • kubelet and kube-proxy
    • network_plugin (such as Calico or Weave)
    • kube-apiserver, kube-scheduler, and kube-controller-manager
    • Add-ons (such as KubeDNS)

    Upgrade considerations

    Kubespray supports rotating certificates used for etcd and Kubernetescomponents, but some manual steps may be required. If you have a pod thatrequires use of a service token and is deployed in a namespace other thankube-system, you will need to manually delete the affected pods afterrotating certificates. This is because all service account tokens are dependenton the apiserver token that is used to generate them. When the certificaterotates, all service account tokens must be rotated as well. During thekubernetes-apps/rotate_tokens role, only pods in kube-system are destroyed andrecreated. All other invalidated service account tokens are cleaned upautomatically, but other pods are not deleted out of an abundance of cautionfor impact to user deployed pods.

    A deployer may want to upgrade specific components in order to minimize riskor save time. This strategy is not covered by CI as of this writing, so it isnot guaranteed to work.

    These commands are useful only for upgrading fully-deployed, healthy, existinghosts. This will definitely not work for undeployed or partially deployedhosts.

    Upgrade docker:

    1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=etcd

    Upgrade vault:

    1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=vault

    Upgrade kubelet:

    Upgrade Kubernetes master components:

    Upgrade network plugins:

    1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=network
    1. ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=apps

    Upgrade just helm (assuming helm_enabled is true):