Vault as the Server TLS Certificate Provider on Kubernetes

Create a Vault policy that authorizes the desired level of access to the secret.

Setup per Consul datacenter

(Added) Configure allowed domains for PKI certificates
Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
Update the Consul on Kubernetes helm chart.

Prior to setting up the data integration between Vault and Consul on Kubernetes, you will need to have:

Read and completed the steps in the section of Vault as a Secrets Backend.
Read the section of Vault as a Secrets Backend.
Complete the section.

First, we need to bootstrap the Vault cluster by enabling and configuring the PKI Secrets Engine to be able to serve TLS certificates to Consul. The process can be as simple as the following, or more complicated such as in this example which also uses an intermediate signing authority.

Enable the PKI Secrets Engine:

Tune the engine to enable longer TTL:

$ vault secrets tune -max-lease-ttl=87600h pki

$ vault secrets tune -max-lease-ttl=87600h pki

Generate the root CA:

Note: The common_name value is comprised of combining global.datacenter dot global.domain.

$ vault write -field=certificate pki/root/generate/internal \
        common_name="dc1.consul" \
        ttl=87600h

$ vault write -field=certificate pki/root/generate/internal \
        common_name="dc1.consul" \
        ttl=87600h

Store the secret in Vault

This step is not valid to this use case because we are not storing a single secret. We are configuring Vault as a provider to mint certificates on an ongoing basis.

To use Vault to issue Server TLS certificates, you will need to create the following:

Vault Policies that will allow the Consul server to access the certificate issuing url.
Vault Policies that will allow the Consul components, e.g. ingress gateways, controller, to access the CA url.

Create Vault Policies for the Server TLS Certificates

Note: The PKI secret path referenced by the Vault Policy below will be your server.serverCert.secretName Helm value.

path "pki/issue/consul-server" {
  capabilities = ["create", "update"]
}

consul-server-policy.hcl

path "pki/issue/consul-server" {
  capabilities = ["create", "update"]
}

Apply the Vault policy by issuing the vault policy write CLI command:

$ vault policy write consul-server consul-server-policy.hcl

$ vault policy write consul-server consul-server-policy.hcl

Create Vault Policies for the CA URL

Next, we will create a policy that allows ["read"] access to the CA URL, this is required for the Consul components to communicate with the Consul servers in order to fetch their auto-encryption certificates.

ca-policy.hcl

path "pki/cert/ca" {
  capabilities = ["read"]
}

$ vault policy write ca-policy ca-policy.hcl

$ vault policy write ca-policy ca-policy.hcl

Note: The PKI secret path referenced by the above Policy will be your global.tls.caCert.secretName Helm value.

Configure allowed domains for PKI certificates

Next, a Vault role for the PKI engine will set the default certificate issuance parameters:

$ vault write pki/roles/consul-server \
    allowed_domains="<Allowed-domains-string>" \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

$ vault write pki/roles/consul-server \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

To generate the <Allowed-domains-string> use the following script as a template:

#!/bin/sh
# NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
export NAME=consulk8s
# NAMESPACE is where the Consul on Kubernetes is installed
export NAMESPACE=consul
# DATACENTER is the value of `global.datacenter` from your Helm values config file
export DATACENTER=dc1
echo allowed_domains=\"$DATACENTER.consul, $NAME-server, $NAME-server.$NAMESPACE, $NAME-server.$NAMESPACE.svc\"

#!/bin/sh
# NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
export NAME=consulk8s
# NAMESPACE is where the Consul on Kubernetes is installed
export NAMESPACE=consul
# DATACENTER is the value of `global.datacenter` from your Helm values config file
export DATACENTER=dc1
echo allowed_domains=\"$DATACENTER.consul, $NAME-server, $NAME-server.$NAMESPACE, $NAME-server.$NAMESPACE.svc\"

Role for Consul servers:

$ vault write auth/kubernetes/role/consul-server \
    bound_service_account_names=<Consul server service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=consul-server \
    ttl=1h

$ vault write auth/kubernetes/role/consul-server \
    bound_service_account_names=<Consul server service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=consul-server \
    ttl=1h

To find out the service account name of the Consul server, you can run:

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/server-serviceaccount.yaml hashicorp/consul

Role for Consul clients:

$ vault write auth/kubernetes/role/consul-client \
    bound_service_account_names=<Consul client service account>  \
    bound_service_account_namespaces=default \
    policies=ca-policy \
    ttl=1h

$ vault write auth/kubernetes/role/consul-client \
    bound_service_account_names=<Consul client service account>  \
    bound_service_account_namespaces=default \
    ttl=1h

To find out the service account name of the Consul client, use the command below.

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/client-serviceaccount.yaml hashicorp/consul

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/client-serviceaccount.yaml hashicorp/consul

Role for CA components:

$ vault write auth/kubernetes/role/consul-ca \
    bound_service_account_names="*" \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=ca-policy \

$ vault write auth/kubernetes/role/consul-ca \
    bound_service_account_names="*" \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=ca-policy \
    ttl=1h

The above Vault Roles will now be your Helm values for global.secretsBackend.vault.consulServerRole and global.secretsBackend.vault.consulCARole respectively.

Update the Consul on Kubernetes helm chart

Now that we’ve configured Vault, you can configure the Consul Helm chart to use the Server TLS certificates from Vault:

global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
  tls:
    enableAutoEncrypt: true
    enabled: true
    caCert:
      secretName: "pki/cert/ca"
server:
  serverCert:
    secretName: "pki/issue/consul-server"
  extraVolumes:
    - type: "secret"
      name: <vaultCASecret>
      load: "false"

values.yaml

global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: consul-server
      consulClientRole: consul-client
      consulCARole: consul-ca
  tls:
    enableAutoEncrypt: true
    enabled: true
    caCert:
      secretName: "pki/cert/ca"
server:
  serverCert:
    secretName: "pki/issue/consul-server"
  extraVolumes:
    - type: "secret"
      name: <vaultCASecret>

$ kubectl create secret generic vault-ca --from-file vault.ca=/path/to/your/vault/