Secrets

    Because Secrets can be created independently of the Pods that use them, there is less risk of the Secret (and its data) being exposed during the workflow of creating, viewing, and editing Pods. Kubernetes, and applications that run in your cluster, can also take additional precautions with Secrets, such as avoiding writing confidential data to nonvolatile storage.

    Secrets are similar to ConfigMaps but are specifically intended to hold confidential data.

    Caution:

    Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.

    In order to safely use Secrets, take at least the following steps:

    1. for Secrets.
    2. Enable or configure RBAC rules that restrict reading data in Secrets (including via indirect means).
    3. Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.

    To use a Secret, a Pod needs to reference the Secret. A Secret can be used with a Pod in three ways:

    The Kubernetes control plane also uses Secrets; for example, are a mechanism to help automate node registration.

    The name of a Secret object must be a valid DNS subdomain name. You can specify the and/or the stringData field when creating a configuration file for a Secret. The data and the stringData fields are optional. The values for all keys in the data field have to be base64-encoded strings. If the conversion to base64 string is not desirable, you can choose to specify the stringData field instead, which accepts arbitrary strings as values.

    The keys of data and stringData must consist of alphanumeric characters, -, _ or .. All key-value pairs in the stringData field are internally merged into the data field. If a key appears in both the data and the stringData field, the value specified in the stringData field takes precedence.

    Types of Secret

    When creating a Secret, you can specify its type using the type field of a Secret resource, or certain equivalent kubectl command line flags (if available). The type of a Secret is used to facilitate programmatic handling of different kinds of confidential data.

    Kubernetes provides several builtin types for some common usage scenarios. These types vary in terms of the validations performed and the constraints Kubernetes imposes on them.

    You can define and use your own Secret type by assigning a non-empty string as the type value for a Secret object. An empty string is treated as an Opaque type. Kubernetes doesn’t impose any constraints on the type name. However, if you are using one of the builtin types, you must meet all the requirements defined for that type.

    Opaque is the default Secret type if omitted from a Secret configuration file. When you create a Secret using kubectl, you will use the generic subcommand to indicate an Opaque Secret type. For example, the following command creates an empty Secret of type Opaque.

    The output looks like:

    1. NAME           TYPE     DATA   AGE
    2. empty-secret   Opaque   0      2m6s

    The DATA column shows the number of data items stored in the Secret. In this case, 0 means we have created an empty Secret.

    Service account token Secrets

    A kubernetes.io/service-account-token type of Secret is used to store a token that identifies a service account. When using this Secret type, you need to ensure that the kubernetes.io/service-account.name annotation is set to an existing service account name. A Kubernetes controller fills in some other fields such as the kubernetes.io/service-account.uid annotation and the token key in the data field set to actual token content.

    The following example configuration declares a service account token Secret:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: secret-sa-sample
    5.   annotations:
    6.     kubernetes.io/service-account.name: "sa-name"
    7. type: kubernetes.io/service-account-token
    8. data:
    9.   # You can include additional key value pairs as you do with Opaque Secrets
    10.   extra: YmFyCg==

    When creating a Pod, Kubernetes automatically creates a service account Secret and automatically modifies your Pod to use this Secret. The service account token Secret contains credentials for accessing the API.

    The automatic creation and use of API credentials can be disabled or overridden if desired. However, if all you need to do is securely access the API server, this is the recommended workflow.

    See the documentation for more information on how service accounts work. You can also check the automountServiceAccountToken field and the serviceAccountName field of the Pod for information on referencing service account from Pods.

    Docker config Secrets

    You can use one of the following type values to create a Secret to store the credentials for accessing a Docker registry for images.

    • kubernetes.io/dockercfg
    • kubernetes.io/dockerconfigjson

    The kubernetes.io/dockercfg type is reserved to store a serialized ~/.dockercfg which is the legacy format for configuring Docker command line. When using this Secret type, you have to ensure the Secret data field contains a .dockercfg key whose value is content of a ~/.dockercfg file encoded in the base64 format.

    The kubernetes.io/dockerconfigjson type is designed for storing a serialized JSON that follows the same format rules as the ~/.docker/config.json file which is a new format for ~/.dockercfg. When using this Secret type, the data field of the Secret object must contain a .dockerconfigjson key, in which the content for the ~/.docker/config.json file is provided as a base64 encoded string.

    Below is an example for a kubernetes.io/dockercfg type of Secret:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: secret-dockercfg
    5. type: kubernetes.io/dockercfg
    6. data:
    7.   .dockercfg: |
    8.         "<base64 encoded ~/.dockercfg file>"

    Note: If you do not want to perform the base64 encoding, you can choose to use the stringData field instead.

    When you create these types of Secrets using a manifest, the API server checks whether the expected key does exists in the data field, and it verifies if the value provided can be parsed as a valid JSON. The API server doesn’t validate if the JSON actually is a Docker config file.

    When you do not have a Docker config file, or you want to use kubectl to create a Docker registry Secret, you can do:

    1. kubectl create secret docker-registry secret-tiger-docker \
    2.   --docker-username=tiger \
    3.   --docker-password=pass113 \
    4.   --docker-email=tiger@acme.com \
    5.   --docker-server=my-registry.example:5000

    This command creates a Secret of type kubernetes.io/dockerconfigjson. If you dump the .dockerconfigjson content from the data field, you will get the following JSON content which is a valid Docker configuration created on the fly:

    1. {
    2.     "apiVersion": "v1",
    3.     "data": {
    4.         ".dockerconfigjson": "eyJhdXRocyI6eyJteS1yZWdpc3RyeTo1MDAwIjp7InVzZXJuYW1lIjoidGlnZXIiLCJwYXNzd29yZCI6InBhc3MxMTMiLCJlbWFpbCI6InRpZ2VyQGFjbWUuY29tIiwiYXV0aCI6ImRHbG5aWEk2Y0dGemN6RXhNdz09In19fQ=="
    5.     },
    6.     "kind": "Secret",
    7.     "metadata": {
    8.         "creationTimestamp": "2021-07-01T07:30:59Z",
    9.         "name": "secret-tiger-docker",
    10.         "namespace": "default",
    11.         "resourceVersion": "566718",
    12.         "uid": "e15c1d7b-9071-4100-8681-f3a7a2ce89ca"
    13.     },
    14.     "type": "kubernetes.io/dockerconfigjson"
    15. }

    Basic authentication Secret

    The kubernetes.io/basic-auth type is provided for storing credentials needed for basic authentication. When using this Secret type, the data field of the Secret must contain one of the following two keys:

    • username: the user name for authentication;
    • password: the password or token for authentication.

    Both values for the above two keys are base64 encoded strings. You can, of course, provide the clear text content using the stringData for Secret creation.

    The following YAML is an example config for a basic authentication Secret:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: secret-basic-auth
    5. type: kubernetes.io/basic-auth
    6. stringData:
    7.   username: admin
    8.   password: t0p-Secret

    The basic authentication Secret type is provided only for user’s convenience. You can create an Opaque for credentials used for basic authentication. However, using the builtin Secret type helps unify the formats of your credentials and the API server does verify if the required keys are provided in a Secret configuration.

    SSH authentication secrets

    The builtin type kubernetes.io/ssh-auth is provided for storing data used in SSH authentication. When using this Secret type, you will have to specify a ssh-privatekey key-value pair in the data (or stringData) field as the SSH credential to use.

    The following YAML is an example config for a SSH authentication Secret:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: secret-ssh-auth
    5. type: kubernetes.io/ssh-auth
    6. data:
    7.   # the data is abbreviated in this example
    8.   ssh-privatekey: |
    9.           MIIEpQIBAAKCAQEAulqb/Y ...

    The SSH authentication Secret type is provided only for user’s convenience. You can create an Opaque for credentials used for SSH authentication. However, using the builtin Secret type helps unify the formats of your credentials and the API server does verify if the required keys are provided in a Secret configuration.

    Caution: SSH private keys do not establish trusted communication between an SSH client and host server on their own. A secondary means of establishing trust is needed to mitigate “man in the middle” attacks, such as a known_hosts file added to a ConfigMap.

    TLS secrets

    Kubernetes provides a builtin Secret type kubernetes.io/tls for storing a certificate and its associated key that are typically used for TLS . This data is primarily used with TLS termination of the Ingress resource, but may be used with other resources or directly by a workload. When using this type of Secret, the tls.key and the tls.crt key must be provided in the data (or stringData) field of the Secret configuration, although the API server doesn’t actually validate the values for each key.

    The following YAML contains an example config for a TLS Secret:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: secret-tls
    5. type: kubernetes.io/tls
    6. data:
    7.   # the data is abbreviated in this example
    8.   tls.crt: |
    9.         MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
    10.   tls.key: |
    11.         MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ ...

    The TLS Secret type is provided for user’s convenience. You can create an Opaque for credentials used for TLS server and/or client. However, using the builtin Secret type helps ensure the consistency of Secret format in your project; the API server does verify if the required keys are provided in a Secret configuration.

    When creating a TLS Secret using kubectl, you can use the tls subcommand as shown in the following example:

    1. kubectl create secret tls my-tls-secret \
    2.   --cert=path/to/cert/file \
    3.   --key=path/to/key/file

    The public/private key pair must exist beforehand. The public key certificate for --cert must be .PEM encoded (Base64-encoded DER format), and match the given private key for --key. The private key must be in what is commonly called PEM private key format, unencrypted. In both cases, the initial and the last lines from PEM (for example, --------BEGIN CERTIFICATE----- and -------END CERTIFICATE---- for a certificate) are not included.

    Bootstrap token Secrets

    A bootstrap token Secret can be created by explicitly specifying the Secret type to bootstrap.kubernetes.io/token. This type of Secret is designed for tokens used during the node bootstrap process. It stores tokens used to sign well known ConfigMaps.

    A bootstrap token Secret is usually created in the kube-system namespace and named in the form bootstrap-token-<token-id> where <token-id> is a 6 character string of the token ID.

    As a Kubernetes manifest, a bootstrap token Secret might look like the following:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: bootstrap-token-5emitj
    5.   namespace: kube-system
    6. type: bootstrap.kubernetes.io/token
    7. data:
    8.   auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4=
    9.   expiration: MjAyMC0wOS0xM1QwNDozOToxMFo=
    10.   token-id: NWVtaXRq
    11.   token-secret: a3E0Z2lodnN6emduMXAwcg==
    12.   usage-bootstrap-authentication: dHJ1ZQ==
    13.   usage-bootstrap-signing: dHJ1ZQ==

    A bootstrap type Secret has the following keys specified under data:

    • token-id: A random 6 character string as the token identifier. Required.
    • token-secret: A random 16 character string as the actual token secret. Required.
    • description: A human-readable string that describes what the token is used for. Optional.
    • expiration: An absolute UTC time using RFC3339 specifying when the token should be expired. Optional.
    • : A boolean flag indicating additional usage for the bootstrap token.
    • auth-extra-groups: A comma-separated list of group names that will be authenticated as in addition to the system:bootstrappers group.

    The above YAML may look confusing because the values are all in base64 encoded strings. In fact, you can create an identical Secret using the following YAML:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   # Note how the Secret is named
    5.   name: bootstrap-token-5emitj
    6.   # A bootstrap token Secret usually resides in the kube-system namespace
    7.   namespace: kube-system
    8. type: bootstrap.kubernetes.io/token
    9. stringData:
    10.   auth-extra-groups: "system:bootstrappers:kubeadm:default-node-token"
    11.   # This token ID is used in the name
    12.   token-id: "5emitj"
    13.   token-secret: "kq4gihvszzgn1p0r"
    14.   # This token can be used for authentication
    15.   usage-bootstrap-authentication: "true"
    16.   # and it can be used for signing
    17.   usage-bootstrap-signing: "true"

    Creating a Secret

    An existing Secret may be edited with the following command:

    1. kubectl edit secrets mysecret

    This will open the default configured editor and allow for updating the base64 encoded Secret values in the data field:

    1. # Please edit the object below. Lines beginning with a '#' will be ignored,
    2. # and an empty file will abort the edit. If an error occurs while saving this file will be
    3. # reopened with the relevant failures.
    4. #
    5. apiVersion: v1
    6. data:
    7.   username: YWRtaW4=
    8.   password: MWYyZDFlMmU2N2Rm
    9. kind: Secret
    10. metadata:
    11.   annotations:
    12.     kubectl.kubernetes.io/last-applied-configuration: { ... }
    13.   creationTimestamp: 2016-01-22T18:41:56Z
    14.   name: mysecret
    15.   namespace: default
    16.   resourceVersion: "164619"
    17.   uid: cfee02d6-c137-11e5-8d73-42010af00002
    18. type: Opaque

    Using Secrets

    Secrets can be mounted as data volumes or exposed as to be used by a container in a Pod. Secrets can also be used by other parts of the system, without being directly exposed to the Pod. For example, Secrets can hold credentials that other parts of the system should use to interact with external systems on your behalf.

    To consume a Secret in a volume in a Pod:

    1. Create a secret or use an existing one. Multiple Pods can reference the same secret.
    2. Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].secret.secretName field equal to the name of the Secret object.
    3. Add a .spec.containers[].volumeMounts[] to each container that needs the secret. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the secrets to appear.
    4. Modify your image or command line so that the program looks for files in that directory. Each key in the secret data map becomes the filename under mountPath.

    This is an example of a Pod that mounts a Secret in a volume:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: mypod
    5. spec:
    6.   containers:
    7.   - name: mypod
    8.     image: redis
    9.     volumeMounts:
    10.     - name: foo
    11.       mountPath: "/etc/foo"
    12.       readOnly: true
    13.   volumes:
    14.   - name: foo
    15.     secret:
    16.       secretName: mysecret

    Each Secret you want to use needs to be referred to in .spec.volumes.

    If there are multiple containers in the Pod, then each container needs its own volumeMounts block, but only one .spec.volumes is needed per Secret.

    You can package many files into one secret, or use many secrets, whichever is convenient.

    Projection of Secret keys to specific paths

    You can also control the paths within the volume where Secret keys are projected. You can use the .spec.volumes[].secret.items field to change the target path of each key:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: mypod
    5. spec:
    6.   containers:
    7.   - name: mypod
    8.     image: redis
    9.     volumeMounts:
    10.     - name: foo
    11.       mountPath: "/etc/foo"
    12.       readOnly: true
    13.   volumes:
    14.   - name: foo
    15.     secret:
    16.       secretName: mysecret
    17.       items:
    18.       - key: username
    19.         path: my-group/my-username

    What will happen:

    • username secret is stored under /etc/foo/my-group/my-username file instead of /etc/foo/username.
    • password secret is not projected.

    If .spec.volumes[].secret.items is used, only keys specified in items are projected. To consume all keys from the secret, all of them must be listed in the items field. All listed keys must exist in the corresponding secret. Otherwise, the volume is not created.

    Secret files permissions

    You can set the file access permission bits for a single Secret key. If you don’t specify any permissions, 0644 is used by default. You can also set a default mode for the entire Secret volume and override per key if needed.

    For example, you can specify a default mode like this:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: mypod
    5. spec:
    6.   containers:
    7.   - name: mypod
    8.     image: redis
    9.     volumeMounts:
    10.     - name: foo
    11.       mountPath: "/etc/foo"
    12.   volumes:
    13.   - name: foo
    14.     secret:
    15.       secretName: mysecret
    16.       defaultMode: 0400

    Then, the secret will be mounted on /etc/foo and all the files created by the secret volume mount will have permission 0400.

    Note that the JSON spec doesn’t support octal notation, so use the value 256 for 0400 permissions. If you use YAML instead of JSON for the Pod, you can use octal notation to specify permissions in a more natural way.

    Note if you kubectl exec into the Pod, you need to follow the symlink to find the expected file mode. For example,

    Check the secrets file mode on the pod.

    1. kubectl exec mypod -it sh
    3. cd /etc/foo
    4. ls -l

    The output is similar to this:

    Follow the symlink to find the correct file mode.

    1. cd /etc/foo/..data
    2. ls -l

    The output is similar to this:

    1. total 8
    2. -r-------- 1 root root 12 May 18 00:18 password
    3. -r-------- 1 root root  5 May 18 00:18 username

    You can also use mapping, as in the previous example, and specify different permissions for different files like this:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: mypod
    5. spec:
    6.   containers:
    7.   - name: mypod
    8.     image: redis
    9.     volumeMounts:
    10.     - name: foo
    11.       mountPath: "/etc/foo"
    12.   volumes:
    13.   - name: foo
    14.     secret:
    15.       secretName: mysecret
    16.       items:
    17.       - key: username
    18.         path: my-group/my-username
    19.         mode: 0777

    In this case, the file resulting in /etc/foo/my-group/my-username will have permission value of 0777. If you use JSON, owing to JSON limitations, you must specify the mode in decimal notation, 511.

    Note that this permission value might be displayed in decimal notation if you read it later.

    Consuming Secret values from volumes

    Inside the container that mounts a secret volume, the secret keys appear as files and the secret values are base64 decoded and stored inside these files. This is the result of commands executed inside the container from the example above:

    1. ls /etc/foo/

    The output is similar to:

    1. username
    2. password
    1. cat /etc/foo/username

    The output is similar to:

    1. admin
    1. cat /etc/foo/password

    The output is similar to:

    1. 1f2d1e2e67df

    The program in a container is responsible for reading the secrets from the files.

    Mounted Secrets are updated automatically

    When a secret currently consumed in a volume is updated, projected keys are eventually updated as well. The kubelet checks whether the mounted secret is fresh on every periodic sync. However, the kubelet uses its local cache for getting the current value of the Secret. The type of the cache is configurable using the ConfigMapAndSecretChangeDetectionStrategy field in the KubeletConfiguration struct. A Secret can be either propagated by watch (default), ttl-based, or by redirecting all requests directly to the API server. As a result, the total delay from the moment when the Secret is updated to the moment when new keys are projected to the Pod can be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type (it equals to watch propagation delay, ttl of cache, or zero correspondingly).

    Note: A container using a Secret as a volume mount will not receive Secret updates.

    Using Secrets as environment variables

    To use a secret in an in a Pod:

    1. Create a secret or use an existing one. Multiple Pods can reference the same secret.
    2. Modify your Pod definition in each container that you wish to consume the value of a secret key to add an environment variable for each secret key you wish to consume. The environment variable that consumes the secret key should populate the secret’s name and key in env[].valueFrom.secretKeyRef.
    3. Modify your image and/or command line so that the program looks for values in the specified environment variables.

    This is an example of a Pod that uses secrets from environment variables:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: secret-env-pod
    5. spec:
    6.   containers:
    7.   - name: mycontainer
    8.     image: redis
    9.     env:
    10.       - name: SECRET_USERNAME
    11.         valueFrom:
    12.           secretKeyRef:
    13.             name: mysecret
    14.             key: username
    15.       - name: SECRET_PASSWORD
    16.         valueFrom:
    17.           secretKeyRef:
    18.             name: mysecret
    19.             key: password
    20.   restartPolicy: Never

    Consuming Secret Values from environment variables

    Inside a container that consumes a secret in the environment variables, the secret keys appear as normal environment variables containing the base64 decoded values of the secret data. This is the result of commands executed inside the container from the example above:

    1. echo $SECRET_USERNAME

    The output is similar to:

    1. admin
    1. echo $SECRET_PASSWORD

    The output is similar to:

    1. 1f2d1e2e67df

    Environment variables are not updated after a secret update

    If a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change.

    Immutable Secrets

    FEATURE STATE: Kubernetes v1.21 [stable]

    The Kubernetes feature Immutable Secrets and ConfigMaps provides an option to set individual Secrets and ConfigMaps as immutable. For clusters that extensively use Secrets (at least tens of thousands of unique Secret to Pod mounts), preventing changes to their data has the following advantages:

    • protects you from accidental (or unwanted) updates that could cause applications outages

    This feature is controlled by the ImmutableEphemeralVolumes , which is enabled by default since v1.19. You can create an immutable Secret by setting the immutable field to true. For example,

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    5. data:
    6.   ...
    7. immutable: true

    Note: Once a Secret or ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data field. You can only delete and recreate the Secret. Existing Pods maintain a mount point to the deleted Secret - it is recommended to recreate these pods.

    Using imagePullSecrets

    The imagePullSecrets field is a list of references to secrets in the same namespace. You can use an imagePullSecrets to pass a secret that contains a Docker (or other) image registry password to the kubelet. The kubelet uses this information to pull a private image on behalf of your Pod. See the for more information about the imagePullSecrets field.

    Manually specifying an imagePullSecret

    You can learn how to specify ImagePullSecrets from the .

    Arranging for imagePullSecrets to be automatically attached

    You can manually create imagePullSecrets, and reference it from a ServiceAccount. Any Pods created with that ServiceAccount or created with that ServiceAccount by default, will get their imagePullSecrets field set to that of the service account. See for a detailed explanation of that process.

    Restrictions

    Secret volume sources are validated to ensure that the specified object reference actually points to an object of type Secret. Therefore, a secret needs to be created before any Pods that depend on it.

    Secret resources reside in a . Secrets can only be referenced by Pods in that same namespace.

    Individual secrets are limited to 1MiB in size. This is to discourage creation of very large secrets which would exhaust the API server and kubelet memory. However, creation of many smaller secrets could also exhaust memory. More comprehensive limits on memory usage due to secrets is a planned feature.

    Secrets must be created before they are consumed in Pods as environment variables unless they are marked as optional. References to secrets that do not exist will prevent the Pod from starting.

    References (secretKeyRef field) to keys that do not exist in a named Secret will prevent the Pod from starting.

    Secrets used to populate environment variables by the envFrom field that have keys that are considered invalid environment variable names will have those keys skipped. The Pod will be allowed to start. There will be an event whose reason is InvalidVariableNames and the message will contain the list of invalid keys that were skipped. The example shows a pod which refers to the default/mysecret that contains 2 invalid keys: 1badkey and 2alsobad.

    1. kubectl get events

    The output is similar to:

    1. LASTSEEN   FIRSTSEEN   COUNT     NAME            KIND      SUBOBJECT                         TYPE      REASON
    2. 0s         0s          1         dapi-test-pod   Pod                                         Warning   InvalidEnvironmentVariableNames   kubelet, 127.0.0.1      Keys [1badkey, 2alsobad] from the EnvFrom secret default/mysecret were skipped since they are considered invalid environment variable names.

    Secret and Pod lifetime interaction

    When a Pod is created by calling the Kubernetes API, there is no check if a referenced secret exists. Once a Pod is scheduled, the kubelet will try to fetch the secret value. If the secret cannot be fetched because it does not exist or because of a temporary lack of connection to the API server, the kubelet will periodically retry. It will report an event about the Pod explaining the reason it is not started yet. Once the secret is fetched, the kubelet will create and mount a volume containing it. None of the Pod’s containers will start until all the Pod’s volumes are mounted.

    Use cases

    Use-Case: As container environment variables

    Create a secret

    Create the Secret:

    1. kubectl apply -f mysecret.yaml

    Use envFrom to define all of the Secret’s data as container environment variables. The key from the Secret becomes the environment variable name in the Pod.

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: secret-test-pod
    5. spec:
    6.   containers:
    7.     - name: test-container
    8.       image: k8s.gcr.io/busybox
    9.       command: [ "/bin/sh", "-c", "env" ]
    10.       envFrom:
    11.       - secretRef:
    12.           name: mysecret
    13.   restartPolicy: Never

    Create a secret containing some ssh keys:

    1. kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=/path/to/.ssh/id_rsa --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub

    The output is similar to:

    1. secret "ssh-key-secret" created

    You can also create a kustomization.yaml with a secretGenerator field containing ssh keys.

    Caution: Think carefully before sending your own ssh keys: other users of the cluster may have access to the secret. Use a service account which you want to be accessible to all the users with whom you share the Kubernetes cluster, and can revoke this account if the users are compromised.

    Now you can create a Pod which references the secret with the ssh key and consumes it in a volume:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: secret-test-pod
    5.   labels:
    6.     name: secret-test
    7. spec:
    8.   volumes:
    9.   - name: secret-volume
    10.     secret:
    11.       secretName: ssh-key-secret
    12.   containers:
    13.   - name: ssh-test-container
    14.     image: mySshImage
    15.     volumeMounts:
    16.     - name: secret-volume
    17.       readOnly: true
    18.       mountPath: "/etc/secret-volume"

    When the container’s command runs, the pieces of the key will be available in:

    1. /etc/secret-volume/ssh-publickey
    2. /etc/secret-volume/ssh-privatekey

    The container is then free to use the secret data to establish an ssh connection.

    Use-Case: Pods with prod / test credentials

    This example illustrates a Pod which consumes a secret containing production credentials and another Pod which consumes a secret with test environment credentials.

    You can create a kustomization.yaml with a secretGenerator field or run kubectl create secret.

    1. kubectl create secret generic prod-db-secret --from-literal=username=produser --from-literal=password=Y4nys7f11

    The output is similar to:

    1. secret "prod-db-secret" created

    You can also create a secret for test environment credentials.

    1. kubectl create secret generic test-db-secret --from-literal=username=testuser --from-literal=password=iluvtests

    The output is similar to:

    1. secret "test-db-secret" created

    Note:

    Special characters such as $, \, *, =, and ! will be interpreted by your shell) and require escaping. In most shells, the easiest way to escape the password is to surround it with single quotes ('). For example, if your actual password is S!B\*d$zDsb=, you should execute the command this way:

    1. kubectl create secret generic dev-db-secret --from-literal=username=devuser --from-literal=password='S!B\*d$zDsb='

    You do not need to escape special characters in passwords from files (--from-file).

    Now make the Pods:

    1. cat <<EOF > pod.yaml
    2. apiVersion: v1
    3. kind: List
    4. items:
    5. - kind: Pod
    6.   apiVersion: v1
    7.   metadata:
    8.     name: prod-db-client-pod
    9.     labels:
    10.       name: prod-db-client
    11.   spec:
    12.     volumes:
    13.     - name: secret-volume
    14.       secret:
    15.         secretName: prod-db-secret
    16.     containers:
    17.     - name: db-client-container
    18.       image: myClientImage
    19.       volumeMounts:
    20.       - name: secret-volume
    21.         readOnly: true
    22.         mountPath: "/etc/secret-volume"
    23. - kind: Pod
    24.   apiVersion: v1
    25.   metadata:
    26.     name: test-db-client-pod
    27.     labels:
    28.       name: test-db-client
    29.   spec:
    30.     volumes:
    31.     - name: secret-volume
    32.       secret:
    33.         secretName: test-db-secret
    34.     containers:
    35.     - name: db-client-container
    36.       image: myClientImage
    37.       volumeMounts:
    38.       - name: secret-volume
    39.         readOnly: true
    40.         mountPath: "/etc/secret-volume"
    41. EOF

    Add the pods to the same kustomization.yaml:

    1. cat <<EOF >> kustomization.yaml
    2. resources:
    3. - pod.yaml
    4. EOF

    Apply all those objects on the API server by running:

    1. kubectl apply -k .

    Both containers will have the following files present on their filesystems with the values for each container’s environment:

    1. /etc/secret-volume/username
    2. /etc/secret-volume/password

    Note how the specs for the two Pods differ only in one field; this facilitates creating Pods with different capabilities from a common Pod template.

    You could further simplify the base Pod specification by using two service accounts:

    1. prod-user with the prod-db-secret
    2. test-user with the test-db-secret

    The Pod specification is shortened to:

    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4.   name: prod-db-client-pod
    5.   labels:
    6.     name: prod-db-client
    7. spec:
    8.   serviceAccount: prod-db-client
    9.   containers:
    10.   - name: db-client-container
    11.     image: myClientImage

    Use-case: dotfiles in a secret volume

    You can make your data “hidden” by defining a key that begins with a dot. This key represents a dotfile or “hidden” file. For example, when the following secret is mounted into a volume, secret-volume:

    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4.   name: dotfile-secret
    5. data:
    6.   .secret-file: dmFsdWUtMg0KDQo=
    7. ---
    8. apiVersion: v1
    9. kind: Pod
    10. metadata:
    11.   name: secret-dotfiles-pod
    12. spec:
    13.   volumes:
    14.   - name: secret-volume
    15.     secret:
    16.       secretName: dotfile-secret
    17.   containers:
    18.   - name: dotfile-test-container
    19.     image: k8s.gcr.io/busybox
    20.     command:
    21.     - ls
    22.     - "-l"
    23.     - "/etc/secret-volume"
    24.     volumeMounts:
    25.     - name: secret-volume
    26.       readOnly: true
    27.       mountPath: "/etc/secret-volume"

    The volume will contain a single file, called .secret-file, and the dotfile-test-container will have this file present at the path /etc/secret-volume/.secret-file.

    Note: Files beginning with dot characters are hidden from the output of ls -l; you must use ls -la to see them when listing directory contents.

    Use-case: Secret visible to one container in a Pod

    Consider a program that needs to handle HTTP requests, do some complex business logic, and then sign some messages with an HMAC. Because it has complex application logic, there might be an unnoticed remote file reading exploit in the server, which could expose the private key to an attacker.

    This could be divided into two processes in two containers: a frontend container which handles user interaction and business logic, but which cannot see the private key; and a signer container that can see the private key, and responds to simple signing requests from the frontend (for example, over localhost networking).

    With this partitioned approach, an attacker now has to trick the application server into doing something rather arbitrary, which may be harder than getting it to read a file.

    Best practices

    Clients that use the Secret API

    When deploying applications that interact with the Secret API, you should limit access using such as RBAC.

    Secrets often hold values that span a spectrum of importance, many of which can cause escalations within Kubernetes (e.g. service account tokens) and to external systems. Even if an individual app can reason about the power of the Secrets it expects to interact with, other apps within the same namespace can render those assumptions invalid.

    For these reasons watch and list requests for secrets within a namespace are extremely powerful capabilities and should be avoided, since listing secrets allows the clients to inspect the values of all secrets that are in that namespace. The ability to watch and list all secrets in a cluster should be reserved for only the most privileged, system-level components.

    Applications that need to access the Secret API should perform get requests on the secrets they need. This lets administrators restrict access to all secrets while that the app needs.

    For improved performance over a looping get, clients can design resources that reference a secret then watch the resource, re-requesting the secret when the reference changes. Additionally, a “bulk watch” API to let clients watch individual resources has also been proposed, and will likely be available in future releases of Kubernetes.

    Protections

    Because secrets can be created independently of the Pods that use them, there is less risk of the secret being exposed during the workflow of creating, viewing, and editing Pods. The system can also take additional precautions with Secrets, such as avoiding writing them to disk where possible.

    A secret is only sent to a node if a Pod on that node requires it. The kubelet stores the secret into a tmpfs so that the secret is not written to disk storage. Once the Pod that depends on the secret is deleted, the kubelet will delete its local copy of the secret data as well.

    There may be secrets for several Pods on the same node. However, only the secrets that a Pod requests are potentially visible within its containers. Therefore, one Pod does not have access to the secrets of another Pod.

    There may be several containers in a Pod. However, each container in a Pod has to request the secret volume in its volumeMounts for it to be visible within the container. This can be used to construct useful security partitions at the Pod level.

    On most Kubernetes distributions, communication between users and the API server, and from the API server to the kubelets, is protected by SSL/TLS. Secrets are protected when transmitted over these channels.

    You can enable for secret data, so that the secrets are not stored in the clear into etcd.

    Risks

    • In the API server, secret data is stored in etcd; therefore:
      • Administrators should enable encryption at rest for cluster data (requires v1.13 or later).
      • Administrators should limit access to etcd to admin users.
      • Administrators may want to wipe/shred disks used by etcd when no longer in use.
      • If running etcd in a cluster, administrators should make sure to use SSL/TLS for etcd peer-to-peer communication.
    • If you configure the secret through a manifest (JSON or YAML) file which has the secret data encoded as base64, sharing this file or checking it in to a source repository means the secret is compromised. Base64 encoding is not an encryption method and is considered the same as plain text.
    • Applications still need to protect the value of secret after reading it from the volume, such as not accidentally logging it or transmitting it to an untrusted party.
    • A user who can create a Pod that uses a secret can also see the value of that secret. Even if the API server policy does not allow that user to read the Secret, the user could run a Pod which exposes the secret.

    What’s next