Vault as the Controller and Connect Inject Webhook Certificate Provider on Kubernetes

In a Consul Helm chart configuration that does not use Vault, webhook-cert-manager normally fulfills the role of ensuring that a valid certificate is updated to the of either controller or connect inject to ensure that Kubernetes can communicate with each of these services.

When Vault is configured as the controller and connect inject Webhook Certificate Provider on Kubernetes:

webhook-cert-manager is no longer deployed to the cluster.
controller and connect inject each get their webhook certificates from its own Vault PKI mount via the injected Vault Agent.
controller and connect inject each need to be configured with its own Vault Role that has necessary permissions to receive certificates from its respective PKI mount.
controller and connect inject each locally update its own mutatingwebhookconfiguration so that Kubernetes can relay events.
Vault manages certificate rotation and rotates certificates to each webhook.

To use Vault as the controller and connect inject Webhook Certificate Provider, we will need to modify the steps outlined in the section:

Create a Vault policy that authorizes the desired level of access to the secret.
(Added) Create Vault PKI roles for controller and connect inject each that establish the domains that each is allowed to issue certificates for.
Create Vault Kubernetes auth roles that link the policy to each Consul on Kubernetes service account that requires access.
Configure the Vault Kubernetes auth roles in the Consul on Kubernetes helm chart.

Complete the following prerequisites prior to implementing the integration described in this topic:

Verify that you have completed the steps described in Systems Integration section of .
You should be familiar with the Data Integration Overview section of .
Configure Vault as the Server TLS Certificate Provider on Kubernetes
Configure
Complete the Bootstrapping the PKI Engine for Controller and Connect Inject Webhooks section.

Bootstrapping the PKI Engine for Controller and Connect Inject Webhooks

The first step is to bootstrap the Vault cluster. Issue the following commands to enable and configure the PKI Secrets Engine to serve TLS certificates for the controller and connect inject webhooks:

Mount the PKI Secrets Engine for each:

$ vault secrets enable -path=controller pki

$ vault secrets enable -path=connect-inject pki

$ vault secrets enable -path=connect-inject pki

Tune the engine mounts to enable longer TTL:

$ vault secrets tune -max-lease-ttl=87600h controller

$ vault secrets tune -max-lease-ttl=87600h controller

$ vault secrets tune -max-lease-ttl=87600h connect-inject

$ vault secrets tune -max-lease-ttl=87600h connect-inject

Generate the root CA for each:

$ vault write -field=certificate controller/root/generate/internal \
        common_name="<helm release name>-controller-webhook" \
        ttl=87600h

$ vault write -field=certificate controller/root/generate/internal \
        common_name="<helm release name>-controller-webhook" \
        ttl=87600h

$ vault write -field=certificate connect-inject/root/generate/internal \
        common_name="<helm release name>-connect-injector" \
        ttl=87600h

$ vault write -field=certificate connect-inject/root/generate/internal \
        common_name="<helm release name>-connect-injector" \
        ttl=87600h

To use Vault to issue controller or connect inject webhook certificates, you will need to create the Vault policies that will allow either controller or connect inject to access its respective certificate-issuing URL.

Create Vault Policies for the Controller and Connect Inject Webhook Certificates

Note: The PKI secret paths referenced by the Vault Policies below will be your global.secretsBackend.vault.controller.tlsCert.secretName and global.secretsBackend.vault.connectInject.tlsCert.secretName Helm values respectively.

The next step is to create a policy that allows ["create", "update"] access to the so Consul controller and connect inject can fetch a new certificate/key pair and provide it to the Kubernetes mutatingwebhookconfiguration. Issue the following commands to create the policy:

$ vault policy write controller-tls-policy - <<EOF
path controller/issue/controller-role {
  capabilities = ["create", "update"]
}
EOF

$ vault policy write connect-inject-policy - <<EOF
path connect-inject/issue/connect-inject-role {
  capabilities = ["create", "update"]
}
EOF

$ vault policy write connect-inject-policy - <<EOF
path connect-inject/issue/connect-inject-role {
  capabilities = ["create", "update"]
}
EOF

Create Vault Policies for the CA URL

Note: The PKI secret paths referenced by the Vault Policies below will be your global.secretsBackend.vault.controller.caCert.secretName and global.secretsBackend.vault.connectInject.caCert.secretName Helm values respectively.

Next, create a policy that allows ["read"] access to the . The policy is required so that Consul components can communicate with the Consul servers in order to fetch their auto-encryption certificates. Issue the following commands to create the policy:

$ vault policy write controller-ca-policy - <<EOF
path controller/cert/ca {
  capabilities = ["read"]
}
EOF

$ vault policy write controller-ca-policy - <<EOF
path controller/cert/ca {
  capabilities = ["read"]
}

$ vault policy write connect-inject-ca-policy - <<EOF
path connect-inject/cert/ca {
  capabilities = ["read"]
}
EOF

$ vault policy write connect-inject-ca-policy - <<EOF
path connect-inject/cert/ca {
  capabilities = ["read"]
}
EOF

Configure allowed domains for PKI certificates

Issue the following command to create a Vault role for the controller PKI engine and set the default parameters for issuing certificates:

$ vault write controller/roles/controller-role \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    generate_lease=true \
    max_ttl="720h"

$ vault write controller/roles/controller-role \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

Issue the following command to create a Vault role for the connect inject PKI engine and set the default parameters for issuing certificates:

$ vault write connect-inject/roles/connect-inject-role \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

$ vault write connect-inject/roles/connect-inject-role \
    allowed_domains="<Allowed-domains-string>" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_localhost=true \
    generate_lease=true \
    max_ttl="720h"

To generate the <Allowed-domains-string> for each use the following script as a template:

#!/bin/sh
# NAME is set to either the value from `global.name` from your Consul K8s value file, or your $HELM_RELEASE_NAME-consul
export NAME=consulk8s
# NAMESPACE is where the Consul on Kubernetes is installed
export NAMESPACE=consul
# DATACENTER is the value of `global.datacenter` from your Helm values config file
export DATACENTER=dc1
echo allowed_domains_controller=\"${NAME}-controller-webhook,${NAME}-controller-webhook.${NAMESPACE},${NAME}-controller-webhook.${NAMESPACE}.svc,${NAME}-controller-webhook.${NAMESPACE}.svc.cluster.local\""
echo allowed_domains_connect_inject=\"${NAME}-connect-injector,${NAME}-connect-injector.${NAMESPACE},${NAME}-connect-injector.${NAMESPACE}.svc,${NAME}-connect-injector.${NAMESPACE}.svc.cluster.local\""

Finally, Kubernetes auth roles need to be created for controller and connect inject webhooks.

Role for Consul controller webhooks:

$ vault write auth/kubernetes/role/controller-role \
    bound_service_account_names=<Consul controller service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=controller-ca-policy \
    ttl=1h

$ vault write auth/kubernetes/role/controller-role \
    bound_service_account_names=<Consul controller service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=controller-ca-policy \
    ttl=1h

To find out the service account name of the Consul controller, you can run:

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/controller-serviceaccount.yaml hashicorp/consul

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/controller-serviceaccount.yaml hashicorp/consul

Role for Consul connect inject webhooks:

$ vault write auth/kubernetes/role/connect-inject-role \
    bound_service_account_names=<Consul connect inject service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=connect-inject-ca-policy \
    ttl=1h

$ vault write auth/kubernetes/role/connect-inject-role \
    bound_service_account_names=<Consul connect inject service account> \
    bound_service_account_namespaces=<Consul installation namespace> \
    policies=connect-inject-ca-policy \

To find out the service account name of the Consul connect inject, use the command below.

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/connect-inject-serviceaccount.yaml hashicorp/consul

 $ helm template --release-name ${RELEASE_NAME} --show-only templates/connect-inject-serviceaccount.yaml hashicorp/consul

Update the Consul on Kubernetes helm chart

Now that we’ve configured Vault, you can configure the Consul Helm chart to use the Server TLS certificates from Vault:

global:
  secretsBackend:
    vault:
      enabled: true
      consulClientRole: "consul-client"
      consulCARole: "consul-ca"
      controllerRole: "controller-role"
      connectInjectRole: "connect-inject-role"
      controller:
        caCert:
          secretName: "controller/cert/ca"
        tlsCert:
          secretName: "controller/issue/controller-role"
      connectInject:
        caCert:
          secretName: "connect-inject/cert/ca"
        tlsCert:
          secretName: "connect-inject/issue/connect-inject-role"
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: "pki/cert/ca"
server:
  serverCert:
    secretName: "pki/issue/consul-server"
  extraVolumes:
    - type: "secret"
      name: <vaultCASecret>
      load: "false"
connectInject:
  enabled: true
controller:
  enabled: true

values.yaml

global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: "consul-server"
      consulClientRole: "consul-client"
      consulCARole: "consul-ca"
      controllerRole: "controller-role"
      connectInjectRole: "connect-inject-role"
      controller:
        caCert:
          secretName: "controller/cert/ca"
        tlsCert:
          secretName: "controller/issue/controller-role"
      connectInject:
        caCert:
          secretName: "connect-inject/cert/ca"
        tlsCert:
          secretName: "connect-inject/issue/connect-inject-role"
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: "pki/cert/ca"
server:
  serverCert:
    secretName: "pki/issue/consul-server"
  extraVolumes:
    - type: "secret"
      name: <vaultCASecret>
      load: "false"
connectInject:
  enabled: true
controller: