Rotating Gossip Encryption Key

    The following steps should only be performed in the primary datacenter if your Consul clusters are federated. Rotating the gossip encryption in the primary datacenter will automatically rotate the gossip encryption in the secondary datacenters.

    Note: Careful precaution should be taken to prohibit new clients from joining during the gossip encryption rotation process, otherwise the new clients will join the gossip pool without knowledge of the new primary gossip encryption key. In addition, deletion of a gossip encryption key from the keyring should occur only after clients have safely migrated to utilizing the new gossip encryption key for communication.

    1. (Optional) If Consul is installed in a dedicated namespace, set the kubeConfig context to the consul namespace. Otherwise, subsequent commands will need to include -n consul.

    2. Generate a new key and store in safe place for retrieval in the future ( is a recommended option).

      This should generate a new key which can be used as the gossip encryption key. In this example, we will be using Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= as the replacement gossip encryption key.

    3. Add new key to consul keyring.

        1. kubectl exec -it consul-server-0 -- /bin/sh
      1. Note: If ACLs are enabled, export the bootstrap token as the CONSUL_HTTP_TOKEN to perform all consul keyring operations. The bootstrap token can be found in the Kubernetes secret consul-bootstrap-acl-token of the primary datacenter.

        1. export CONSUL_HTTP_TOKEN=<INSERT BOOTSTRAP ACL TOKEN>
      2. Install the new Gossip encryption key with the consul keyring command:

        1. consul keyring -install="Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="
        2. ==> Installing new gossip encryption key...

        Consul automatically propagates this encryption key across all clients and servers across the cluster and the federation if Consul federation is enabled.

      3. List the keys in the keyring to verify the new key has been installed successfully.

    4. Use the new key as the gossip encryption key.

      1. After the new key has been added to the keychain, you can install it as the new gossip encryption key. Run the following command in the Consul Agent pod using kubectl exec:

        1. ==> Changing primary gossip encryption key...
        1. consul keyring -list
        2. WAN:
        3. CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [2/2]
        4. Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [2/2]
        5. dc1 (LAN):
        6. CL6M+jKj3630CZLXI0IRVeyci1jgIAveiZKvdtTybbA= [4/4]
        7. Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]
        8. dc2 (LAN):
        9. Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w= [4/4]
    5. Update the Kubernetes secrets with the latest gossip encryption key.

      Update the gossip encryption Kubernetes Secret with the value of the new gossip encryption key to ensure that subsequent commands execute successfully. The name of the secret that stores the value of the gossip encryption key can be found in the Helm values file:

      1. global:
      2. gossipEncryption:
      3. secretName: consul-gossip-encryption-key
      4. secretKey: key
      1. kubectl patch secret consul-gossip-encryption-key -p '{"stringData":{"key": "Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="}}'

      Note: In the case of federated Consul clusters, update the federation-secret value for the gossip encryption key. The name of the secret and key can be found in the values file of the secondary datacenter.

      1. kubectl patch secret consul-federation -p '{"stringData":{"gossipEncryptionKey": "Wa6/XFAnYy0f9iqVH2iiG+yore3CqHSemUy4AIVTa/w="}}'
    6. Remove the old key once the new one has been installed successfully.

      1. kubectl exec into a Consul Agent pod (server or client) and add the new key to the Consul Keyring. This can be performed by running the following command:

        1. kubectl exec -it consul-server-0 -- /bin/sh
      2. Note: If ACLs are enabled, export the bootstrap token as the CONSUL_HTTP_TOKEN to perform all consul keyring operations.

        1. export CONSUL_HTTP_TOKEN=<INSERT BOOTSTRAP ACL TOKEN>
        1. ==> Removing gossip encryption key...