Manage TLS Certificates in a Cluster

    certificates.k8s.io API uses a protocol that is similar to the ACME draft.

    Note: Certificates created using the certificates.k8s.io API are signed by a . It is possible to configure your cluster to use the cluster root CA for this purpose, but you should never rely on this. Do not assume that these certificates will validate against the cluster root CA.

    You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using or you can use one of these Kubernetes playgrounds:

    You need the cfssl tool. You can download cfssl from https://github.com/cloudflare/cfssl/releases.

    Some steps in this page use the jq tool. If you don’t have jq, you can install it via your operating system’s software sources, or fetch it from .

    Trusting TLS in a cluster

    Trusting the from an application running as a pod usually requires some extra application configuration. You will need to add the CA certificate bundle to the list of CA certificates that the TLS client or server trusts. For example, you would do this with a golang TLS config by parsing the certificate chain and adding the parsed certificates to the RootCAs field in the tls.Config struct.

    Note:

    Even though the custom CA certificate may be included in the filesystem (in the ConfigMap kube-root-ca.crt), you should not use that certificate authority for any purpose other than to verify internal Kubernetes endpoints. An example of an internal Kubernetes endpoint is the Service named kubernetes in the default namespace.

    If you want to use a custom certificate authority for your workloads, you should generate that CA separately, and distribute its CA certificate using a that your pods have access to read.

    Requesting a certificate

    The following section demonstrates how to create a TLS certificate for a Kubernetes service accessed through DNS.

    Note: This tutorial uses CFSSL: Cloudflare’s PKI and TLS toolkit to know more.

    Generate a private key and certificate signing request (or CSR) by running the following command:

    Where 192.0.2.24 is the service’s cluster IP, my-svc.my-namespace.svc.cluster.local is the service’s DNS name, 10.0.34.2 is the pod’s IP and my-pod.my-namespace.pod.cluster.local is the pod’s DNS name. You should see the output similar to:

    1. 2022/02/01 11:45:32 [INFO] generate received request
    2. 2022/02/01 11:45:32 [INFO] received CSR
    3. 2022/02/01 11:45:32 [INFO] generating key: ecdsa-256
    4. 2022/02/01 11:45:32 [INFO] encoded CSR

    This command generates two files; it generates server.csr containing the PEM encoded certification request, and server-key.pem containing the PEM encoded key to the certificate that is still to be created.

    Create a CertificateSigningRequest object to send to the Kubernetes API

    1. cat <<EOF | kubectl apply -f -
    2. apiVersion: certificates.k8s.io/v1
    3. kind: CertificateSigningRequest
    4. metadata:
    5.   name: my-svc.my-namespace
    6. spec:
    7.   request: $(cat server.csr | base64 | tr -d '\n')
    8.   signerName: example.com/serving
    9.   usages:
    10.   - digital signature
    11.   - key encipherment
    12.   - server auth
    13. EOF

    Notice that the server.csr file created in step 1 is base64 encoded and stashed in the .spec.request field. You are also requesting a certificate with the “digital signature”, “key encipherment”, and “server auth” key usages, signed by an example example.com/serving signer. A specific signerName must be requested. View documentation for for more information.

    The CSR should now be visible from the API in a Pending state. You can see it by running:

      1. Name:                   my-svc.my-namespace
      3. Annotations:            <none>
      4. CreationTimestamp:      Tue, 01 Feb 2022 11:49:15 -0500
      5. Requesting User:        yourname@example.com
      6. Signer:                 example.com/serving
      7. Status:                 Pending
      8. Subject:
      9.         Common Name:    my-pod.my-namespace.pod.cluster.local
      10.         Serial Number:
      11. Subject Alternative Names:
      12.         DNS Names:      my-pod.my-namespace.pod.cluster.local
      13.                         my-svc.my-namespace.svc.cluster.local
      14.         IP Addresses:   192.0.2.24
      15.                         10.0.34.2
      16. Events: <none>

      Get the CertificateSigningRequest approved

      Approving the is either done by an automated approval process or on a one off basis by a cluster administrator. If you’re authorized to approve a certificate request, you can do that manually using kubectl; for example:

      1. kubectl certificate approve my-svc.my-namespace
      1. certificatesigningrequest.certificates.k8s.io/my-svc.my-namespace approved

      You should now see the following:

      1. NAME                  AGE   SIGNERNAME            REQUESTOR              REQUESTEDDURATION   CONDITION
      2. my-svc.my-namespace   10m   example.com/serving   yourname@example.com   <none>              Approved

      This means the certificate request has been approved and is waiting for the requested signer to sign it.

      Next, you’ll play the part of a certificate signer, issue the certificate, and upload it to the API.

      A signer would typically watch the CertificateSigningRequest API for objects with its signerName, check that they have been approved, sign certificates for those requests, and update the API object status with the issued certificate.

      You need an authority to provide the digital signature on the new certificate.

      First, create a signing certificate by running the following:

      1. cat <<EOF | cfssl gencert -initca - | cfssljson -bare ca
      2. {
      3.   "CN": "My Example Signer",
      4.   "key": {
      5.     "algo": "rsa",
      6.     "size": 2048
      7.   }
      8. }
      9. EOF

      You should see output similar to:

      1. 2022/02/01 11:50:39 [INFO] generating a new CA key and certificate from CSR
      2. 2022/02/01 11:50:39 [INFO] generate received request
      3. 2022/02/01 11:50:39 [INFO] received CSR
      4. 2022/02/01 11:50:39 [INFO] encoded CSR
      5. 2022/02/01 11:50:39 [INFO] signed certificate with serial number 263983151013686720899716354349605500797834580472

      This produces a certificate authority key file (ca-key.pem) and certificate (ca.pem).

      1. {
      3.         "default": {
      4.             "usages": [
      5.                 "digital signature",
      6.                 "key encipherment",
      7.                 "server auth"
      8.             ],
      9.             "expiry": "876000h",
      10.             "ca_constraint": {
      11.                 "is_ca": false
      12.             }
      13.         }
      14.     }
      15. }

      Use a server-signing-config.json signing configuration and the certificate authority key file and certificate to sign the certificate request:

      1. kubectl get csr my-svc.my-namespace -o jsonpath='{.spec.request}' | \
      2.   base64 --decode | \
      3.   cfssl sign -ca ca.pem -ca-key ca-key.pem -config server-signing-config.json - | \
      4.   cfssljson -bare ca-signed-server

      You should see the output similar to:

      1. 2022/02/01 11:52:26 [INFO] signed certificate with serial number 576048928624926584381415936700914530534472870337

      Finally, populate the signed certificate in the API object’s status:

      Note: This uses the command line tool to populate the base64-encoded content in the .status.certificate field. If you do not have jq, you can also save the JSON output to a file, populate this field manually, and upload the resulting file.

      Once the CSR is approved and the signed certificate is uploaded, run:

      1. kubectl get csr

      The output is similar to:

      1. NAME                  AGE   SIGNERNAME            REQUESTOR              REQUESTEDDURATION   CONDITION
      2. my-svc.my-namespace   20m   example.com/serving   yourname@example.com   <none>              Approved,Issued

      Download the certificate and use it

      Now, as the requesting user, you can download the issued certificate and save it to a server.crt file by running the following:

      1. kubectl get csr my-svc.my-namespace -o jsonpath='{.status.certificate}' \
      2.     | base64 --decode > server.crt

      Now you can populate server.crt and server-key.pem in a that you could later mount into a Pod (for example, to use with a webserver that serves HTTPS).

      1. kubectl create secret tls server --cert server.crt --key server-key.pem
      1. secret/server created

      Finally, you can populate ca.pem into a ConfigMap and use it as the trust root to verify the serving certificate:

      1. kubectl create configmap example-serving-ca --from-file ca.crt=ca.pem

      Approving CertificateSigningRequests

      A Kubernetes administrator (with appropriate permissions) can manually approve (or deny) CertificateSigningRequests by using the kubectl certificate approve and kubectl certificate deny commands. However if you intend to make heavy usage of this API, you might consider writing an automated certificates controller.

      Caution:

      The ability to approve CSRs decides who trusts whom within your environment. The ability to approve CSRs should not be granted broadly or lightly.

      You should make sure that you confidently understand both the verification requirements that fall on the approver and the repercussions of issuing a specific certificate before you grant the approve permission.

      Whether a machine or a human using kubectl as above, the role of the approver is to verify that the CSR satisfies two requirements:

      1. The subject of the CSR controls the private key used to sign the CSR. This addresses the threat of a third party masquerading as an authorized subject. In the above example, this step would be to verify that the pod controls the private key used to generate the CSR.
      2. The subject of the CSR is authorized to act in the requested context. This addresses the threat of an undesired subject joining the cluster. In the above example, this step would be to verify that the pod is allowed to participate in the requested service.

      If and only if these two requirements are met, the approver should approve the CSR and otherwise should deny the CSR.

      For more information on certificate approval and access control, read the Certificate Signing Requests reference page.