How to rotate all secrets / credentials

• “secrets” are symmetric credentials.

• “keypairs” are pairs of X.509 certificates and their corresponding private keys. The exceptions are “service-account” keypairs, which are stored as certificate and private key pairs, but do not use any part of the certificates other than the public keys.

Keypairs are grouped into named “keysets”, according to their use. For example, the “kubernetes-ca” keyset is used for the cluster’s Kubernetes general CA. Each keyset has a single primary keypair, which is the one whose private key is used. The remaining, secondary keypairs are either trusted or distrusted. The trusted keypairs, including the primary keypair, have their certificates included in relevant trust stores.

You may gracefully rotate keypairs of keysets that are either Certificate Authorities or are “service-account” by performing the following procedure. Other keypairs will be automatically reissued by a non-dryrun when their issuing CA is rotated.

Create a new keypair for each keyset that you are going to rotate. Then update the cluster and perform a rolling update. To stage all rotatable keysets, run:

Rollback procedure

A failure at this stage is unlikely. To roll back this change:

• Use kops get keypairs to get the IDs of the newly created keysets.
• Then use kops distrust keypair to distrust each of them by keyset and ID.
• Then use kops update cluster --yes
• Then use kops rolling-update cluster --yes

2. Export and distribute new kubeconfig certificate-authority-data

If you are rotating the Kubernetes general CA (“kubernetes-ca” or “all”) and you are not using a load balancer for the Kubernetes API with its own separate certificate, export a new kubeconfig with the new CA certificate included in the certificate-authority-data field for the cluster:

kops export kubecfg

Distribute the new certificate-authority-data to all clients of that cluster’s Kubernetes API.

Rollback procedure

To roll back this change, distribute the previous kubeconfig certificate-authority-data.

3. Promote the new keypairs

Promote the new keypairs to primary with:

kops promote keypair all
kops update cluster --yes
kops rolling-update cluster --yes

On cloud providers, such as AWS, that use kops-controller to bootstrap worker nodes, after the kops update cluster --yes step there is a temporary impediment to node scale-up. Instances using the new launch template will not be able to bootstrap off of old kops-controllers. Similarly, instances using the old launch template and which have not yet bootstrapped will not be able to bootstrap off of new kops-controllers. The subsequent rolling update will eventually replace all instances using the old launch template.

Rollback procedure

The most likely failure at this stage would be a client of the Kubernetes API that did not get the new certificate-authority-data and thus do not trust the new TLS server certificate.

To roll back this change:

• Use kops get keypairs to get the IDs of the previous primary keysets, most likely by identifying the issue dates.
• Then use kops promote keypair to promote each of them by keyset and ID.
• Then use kops rolling-update cluster --yes

If you are rotating the Kubernetes general CA (“kubernetes-ca” or “all”) and have kubeconfigs with cluster admin credentials, export new kubeconfigs with new admin credentials for the cluster:

Distribute the new credentials to all clients that require them.

Rollback procedure

To roll back this change, distribute the previous kubeconfig admin credentials.

5. Distrust the previous keypairs

Remove trust in the previous keypairs with:

Rollback procedure

The most likely failure at this stage would be a client of the Kubernetes API that is still using a credential issued by the previous keypair.

To roll back this change:

• Use kops get keypairs --distrusted to get the IDs of the previously trusted keysets, most likely by identifying the distrust dates.
• Then use kops trust keypair to trust each of them by keyset and ID.
• Then use kops update cluster --yes
• Then use kops rolling-update cluster --force --yes

6. Export and distribute new kubeconfig certificate-authority-data

If you are rotating the Kubernetes general CA (“kubernetes-ca” or “all”) and you are not using a load balancer for the Kubernetes API with its own separate certificate, export a new kubeconfig with the previous CA certificate removed from the certificate-authority-data field for the cluster:

kops export kubecfg

Distribute the new certificate-authority-data to all clients of that cluster’s Kubernetes API.

Rollback procedure

To roll back this change, distribute the previous kubeconfig certificate-authority-data.

Rotating the API Server encryptionconfig

See for information on how to gracefully rotate keys in the encryptionconfig.

Use kops create secret encryptionconfig --force to update the encryptionconfig secret. Following that, use kops update cluster --yes and kops rolling-update cluster --yes.

See the Cilium documentation for information on how to gracefully rotate the Cilium IPSec keys.

Use kops create secret ciliumpassword --force to update the cilium-ipsec-keys secret. Following that, use kops update cluster --yes and kops rolling-update cluster --yes.

Rotating the Docker secret

[TODO]

Use kops create secret dockerconfig --force to update the Docker secret. Following that, use kops update cluster --yes and .

Use kops create secret weavepassword --force to update the Docker secret. Following that, use kops update cluster --yes and kops rolling-update cluster --cloudonly --yes.

Legacy procedure

The following is the procedure to rotate secrets and keypairs in kOps versions prior to 1.22.

This is a disruptive procedure.

Delete all secrets & keypairs that kOps is holding:

kops get secrets  | grep '^Secret' | awk '{print $2}' | xargs -I {} kops delete secret secret {}
kops get secrets  | grep '^Keypair' | awk '{print $2}' | xargs -I {} kops delete secret keypair {}

2. Recreate all secrets

Now run kops update to regenerate the secrets & keypairs.

kops update cluster
kops update cluster --yes

kOps may fail to recreate all the keys on first try. If you get errors about ca key for ‘ca’ not being found, run kops update cluster --yes once more.

3. Force cluster to use new secrets

Now you will have to remove the etcd certificates from every master.

Find all the master IPs. One easy way of doing that is running

Then SSH into each node and run

sudo find /mnt/ -name server.* | xargs -I {} sudo rm {}
sudo find /mnt/ -name me.* | xargs -I {} sudo rm {}

You need to reboot every node (using a rolling-update). You have to use --cloudonly because the keypair no longer matches.

kops rolling-update cluster --cloudonly --force --yes

Re-export kubecfg with new settings:

kops export kubecfg

Now the service account tokens will need to be regenerated inside the cluster:

kops toolbox dump and find a master IP

Then ssh admin@${IP} and run this to delete all the service account tokens:

5. Verify the cluster is back up