Declarative Management of Kubernetes Objects Using Kustomize

    Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command:

    To apply those Resources, run with --kustomize or -k flag:

    1. kubectl apply -k <kustomization_directory>

    Install .

    You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

    To check the version, enter kubectl version.

    Overview of Kustomize

    Kustomize is a tool for customizing Kubernetes configurations. It has the following features to manage application configuration files:

    • generating resources from other sources
    • setting cross-cutting fields for resources
    • composing and customizing collections of resources

    ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a .properties file or an SSH keyfile. Kustomize has secretGenerator and configMapGenerator, which generate Secret and ConfigMap from files or literals.

    configMapGenerator

    To generate a ConfigMap from a file, add an entry to the files list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a .properties file:

    1. # Create a application.properties file
    2. cat <<EOF >application.properties
    3. FOO=Bar
    4. EOF
    6. cat <<EOF >./kustomization.yaml
    7. configMapGenerator:
    8. - name: example-configmap-1
    9.   files:
    10.   - application.properties
    11. EOF

    The generated ConfigMap can be examined with the following command:

    1. kubectl kustomize ./

    The generated ConfigMap is:

    1. apiVersion: v1
    2. data:
    3.   application.properties: |
    4.         FOO=Bar
    5. kind: ConfigMap
    6. metadata:
    7.   name: example-configmap-1-8mbdf7882g

    To generate a ConfigMap from an env file, add an entry to the envs list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a .env file:

    1. # Create a .env file
    2. cat <<EOF >.env
    3. FOO=Bar
    4. EOF
    6. cat <<EOF >./kustomization.yaml
    7. configMapGenerator:
    8. - name: example-configmap-1
    9.   envs:
    10.   - .env
    11. EOF

    The generated ConfigMap can be examined with the following command:

    1. kubectl kustomize ./

    The generated ConfigMap is:

    1. apiVersion: v1
    2. data:
    3.   FOO: Bar
    4. kind: ConfigMap
    5. metadata:
    6.   name: example-configmap-1-42cfbf598f

    Note: Each variable in the .env file becomes a separate key in the ConfigMap that you generate. This is different from the previous example which embeds a file named .properties (and all its entries) as the value for a single key.

    ConfigMaps can also be generated from literal key-value pairs. To generate a ConfigMap from a literal key-value pair, add an entry to the literals list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a key-value pair:

    1. cat <<EOF >./kustomization.yaml
    2. configMapGenerator:
    3. - name: example-configmap-2
    4.   literals:
    5.   - FOO=Bar
    6. EOF

    The generated ConfigMap can be checked by the following command:

    1. kubectl kustomize ./
    1. apiVersion: v1
    2. data:
    3.   FOO: Bar
    4. kind: ConfigMap
    5. metadata:
    6.   name: example-configmap-2-g2hdhfc6tk

    To use a generated ConfigMap in a Deployment, reference it by the name of the configMapGenerator. Kustomize will automatically replace this name with the generated name.

    This is an example deployment that uses a generated ConfigMap:

    1. # Create a application.properties file
    2. cat <<EOF >application.properties
    3. FOO=Bar
    4. EOF
    6. cat <<EOF >deployment.yaml
    7. apiVersion: apps/v1
    8. kind: Deployment
    9. metadata:
    10.   name: my-app
    11.   labels:
    12.     app: my-app
    13. spec:
    14.   selector:
    15.     matchLabels:
    16.       app: my-app
    17.   template:
    18.     metadata:
    19.       labels:
    20.         app: my-app
    21.     spec:
    22.       containers:
    23.       - name: app
    24.         image: my-app
    25.         volumeMounts:
    26.         - name: config
    27.           mountPath: /config
    28.       volumes:
    29.       - name: config
    30.         configMap:
    31.           name: example-configmap-1
    32. EOF
    34. cat <<EOF >./kustomization.yaml
    35. resources:
    36. - deployment.yaml
    37. configMapGenerator:
    38. - name: example-configmap-1
    39.   files:
    40.   - application.properties
    41. EOF

    Generate the ConfigMap and Deployment:

    1. kubectl kustomize ./

    The generated Deployment will refer to the generated ConfigMap by name:

    secretGenerator

    You can generate Secrets from files or literal key-value pairs. To generate a Secret from a file, add an entry to the files list in secretGenerator. Here is an example of generating a Secret with a data item from a file:

    1. # Create a password.txt file
    2. cat <<EOF >./password.txt
    3. username=admin
    4. password=secret
    5. EOF
    7. cat <<EOF >./kustomization.yaml
    8. secretGenerator:
    9. - name: example-secret-1
    10.   files:
    11.   - password.txt
    12. EOF

    The generated Secret is as follows:

    1. apiVersion: v1
    2. data:
    3.   password.txt: dXNlcm5hbWU9YWRtaW4KcGFzc3dvcmQ9c2VjcmV0Cg==
    4. kind: Secret
    5. metadata:
    6.   name: example-secret-1-t2kt65hgtb
    7. type: Opaque

    To generate a Secret from a literal key-value pair, add an entry to literals list in secretGenerator. Here is an example of generating a Secret with a data item from a key-value pair:

    1. cat <<EOF >./kustomization.yaml
    2. secretGenerator:
    3. - name: example-secret-2
    4.   literals:
    5.   - username=admin
    6.   - password=secret
    7. EOF

    The generated Secret is as follows:

    1. apiVersion: v1
    2. data:
    3.   password: c2VjcmV0
    4.   username: YWRtaW4=
    5. kind: Secret
    6. metadata:
    7.   name: example-secret-2-t52t6g96d8
    8. type: Opaque

    Like ConfigMaps, generated Secrets can be used in Deployments by referring to the name of the secretGenerator:

    1. # Create a password.txt file
    2. cat <<EOF >./password.txt
    3. username=admin
    4. password=secret
    5. EOF
    7. cat <<EOF >deployment.yaml
    8. apiVersion: apps/v1
    9. kind: Deployment
    10. metadata:
    11.   name: my-app
    12.   labels:
    13.     app: my-app
    14. spec:
    15.   selector:
    16.     matchLabels:
    17.       app: my-app
    18.   template:
    19.     metadata:
    20.       labels:
    21.         app: my-app
    22.     spec:
    23.       containers:
    24.       - name: app
    25.         image: my-app
    26.         volumeMounts:
    27.         - name: password
    28.           mountPath: /secrets
    29.       volumes:
    30.       - name: password
    31.         secret:
    32.           secretName: example-secret-1
    33. EOF
    35. cat <<EOF >./kustomization.yaml
    36. resources:
    37. - deployment.yaml
    38. secretGenerator:
    39. - name: example-secret-1
    40.   files:
    41.   - password.txt
    42. EOF

    generatorOptions

    The generated ConfigMaps and Secrets have a content hash suffix appended. This ensures that a new ConfigMap or Secret is generated when the contents are changed. To disable the behavior of appending a suffix, one can use generatorOptions. Besides that, it is also possible to specify cross-cutting options for generated ConfigMaps and Secrets.

    1. cat <<EOF >./kustomization.yaml
    2. configMapGenerator:
    3. - name: example-configmap-3
    4.   literals:
    5.   - FOO=Bar
    6. generatorOptions:
    7.   disableNameSuffixHash: true
    8.   labels:
    9.     type: generated
    10.   annotations:
    11.     note: generated
    12. EOF

    Runkubectl kustomize ./ to view the generated ConfigMap:

    1. apiVersion: v1
    2. data:
    3.   FOO: Bar
    4. kind: ConfigMap
    5. metadata:
    6.   annotations:
    7.     note: generated
    8.   labels:
    9.     type: generated
    10.   name: example-configmap-3

    It is quite common to set cross-cutting fields for all Kubernetes resources in a project. Some use cases for setting cross-cutting fields:

    • adding the same name prefix or suffix
    • adding the same set of labels
    • adding the same set of annotations

    Here is an example:

    1. # Create a deployment.yaml
    2. cat <<EOF >./deployment.yaml
    3. apiVersion: apps/v1
    4. kind: Deployment
    5. metadata:
    6.   name: nginx-deployment
    7.   labels:
    8.     app: nginx
    9. spec:
    11.     matchLabels:
    12.       app: nginx
    13.   template:
    14.     metadata:
    15.       labels:
    16.         app: nginx
    17.     spec:
    18.       containers:
    19.       - name: nginx
    20.         image: nginx
    21. EOF
    23. cat <<EOF >./kustomization.yaml
    24. namespace: my-namespace
    25. namePrefix: dev-
    26. nameSuffix: "-001"
    27. commonLabels:
    28.   app: bingo
    29. commonAnnotations:
    30.   oncallPager: 800-555-1212
    31. resources:
    32. - deployment.yaml
    33. EOF

    Run kubectl kustomize ./ to view those fields are all set in the Deployment Resource:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   annotations:
    5.     oncallPager: 800-555-1212
    6.   labels:
    7.     app: bingo
    8.   name: dev-nginx-deployment-001
    9.   namespace: my-namespace
    10. spec:
    11.   selector:
    12.     matchLabels:
    13.       app: bingo
    14.   template:
    15.     metadata:
    16.       annotations:
    17.         oncallPager: 800-555-1212
    18.       labels:
    19.         app: bingo
    20.     spec:
    21.       containers:
    22.       - image: nginx
    23.         name: nginx

    It is common to compose a set of Resources in a project and manage them inside the same file or directory. Kustomize offers composing Resources from different files and applying patches or other customization to them.

    Composing

    Kustomize supports composition of different resources. The resources field, in the kustomization.yaml file, defines the list of resources to include in a configuration. Set the path to a resource’s configuration file in the resources list. Here is an example of an NGINX application comprised of a Deployment and a Service:

    1. # Create a deployment.yaml file
    2. cat <<EOF > deployment.yaml
    3. apiVersion: apps/v1
    4. kind: Deployment
    5. metadata:
    6.   name: my-nginx
    7. spec:
    8.   selector:
    9.     matchLabels:
    10.       run: my-nginx
    11.   replicas: 2
    12.   template:
    13.     metadata:
    14.       labels:
    15.         run: my-nginx
    16.     spec:
    17.       containers:
    18.       - name: my-nginx
    19.         image: nginx
    20.         ports:
    21.         - containerPort: 80
    22. EOF
    24. # Create a service.yaml file
    25. cat <<EOF > service.yaml
    26. apiVersion: v1
    27. kind: Service
    28. metadata:
    29.   name: my-nginx
    30.   labels:
    31.     run: my-nginx
    32. spec:
    33.   ports:
    34.   - port: 80
    35.     protocol: TCP
    36.   selector:
    37.     run: my-nginx
    38. EOF
    40. # Create a kustomization.yaml composing them
    41. cat <<EOF >./kustomization.yaml
    42. resources:
    43. - deployment.yaml
    44. - service.yaml
    45. EOF

    Customizing

    Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902. patchesStrategicMerge is a list of file paths. Each file should be resolved to a strategic merge patch. The names inside the patches must match Resource names that are already loaded. Small patches that do one thing are recommended. For example, create one patch for increasing the deployment replica number and another patch for setting the memory limit.

    1. # Create a deployment.yaml file
    2. cat <<EOF > deployment.yaml
    3. apiVersion: apps/v1
    4. kind: Deployment
    5. metadata:
    6.   name: my-nginx
    7. spec:
    8.   selector:
    9.     matchLabels:
    10.       run: my-nginx
    11.   replicas: 2
    12.   template:
    13.     metadata:
    14.       labels:
    15.         run: my-nginx
    16.     spec:
    17.       containers:
    18.       - name: my-nginx
    19.         image: nginx
    20.         ports:
    21.         - containerPort: 80
    22. EOF
    24. # Create a patch increase_replicas.yaml
    25. cat <<EOF > increase_replicas.yaml
    26. apiVersion: apps/v1
    27. kind: Deployment
    28. metadata:
    29.   name: my-nginx
    30. spec:
    31.   replicas: 3
    32. EOF
    34. # Create another patch set_memory.yaml
    35. cat <<EOF > set_memory.yaml
    36. apiVersion: apps/v1
    37. kind: Deployment
    38. metadata:
    39.   name: my-nginx
    40. spec:
    41.   template:
    42.     spec:
    43.       containers:
    44.       - name: my-nginx
    45.         resources:
    46.           limits:
    47.             memory: 512Mi
    48. EOF
    50. cat <<EOF >./kustomization.yaml
    51. resources:
    52. - deployment.yaml
    53. patchesStrategicMerge:
    54. - increase_replicas.yaml
    55. - set_memory.yaml
    56. EOF

    Run kubectl kustomize ./ to view the Deployment:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: my-nginx
    5. spec:
    6.   replicas: 3
    7.   selector:
    8.     matchLabels:
    9.       run: my-nginx
    10.   template:
    11.     metadata:
    12.       labels:
    13.         run: my-nginx
    14.     spec:
    15.       containers:
    16.       - image: nginx
    17.         name: my-nginx
    18.         ports:
    19.         - containerPort: 80
    20.         resources:
    21.           limits:
    22.             memory: 512Mi

    Not all Resources or fields support strategic merge patches. To support modifying arbitrary fields in arbitrary Resources, Kustomize offers applying through patchesJson6902. To find the correct Resource for a Json patch, the group, version, kind and name of that Resource need to be specified in kustomization.yaml. For example, increasing the replica number of a Deployment object can also be done through patchesJson6902.

    Run kubectl kustomize ./ to see the replicas field is updated:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: my-nginx
    5. spec:
    6.   replicas: 3
    7.   selector:
    8.     matchLabels:
    9.       run: my-nginx
    10.   template:
    11.     metadata:
    12.       labels:
    13.         run: my-nginx
    14.     spec:
    15.       containers:
    16.       - image: nginx
    17.         name: my-nginx
    18.         ports:
    19.         - containerPort: 80

    In addition to patches, Kustomize also offers customizing container images or injecting field values from other objects into containers without creating patches. For example, you can change the image used inside containers by specifying the new image in images field in kustomization.yaml.

    1. cat <<EOF > deployment.yaml
    2. apiVersion: apps/v1
    3. kind: Deployment
    4. metadata:
    5.   name: my-nginx
    6. spec:
    7.   selector:
    8.     matchLabels:
    9.       run: my-nginx
    10.   template:
    11.     metadata:
    12.       labels:
    13.         run: my-nginx
    14.     spec:
    15.       containers:
    16.       - name: my-nginx
    18.         ports:
    19.         - containerPort: 80
    20. EOF
    22. cat <<EOF >./kustomization.yaml
    23. resources:
    24. - deployment.yaml
    25. images:
    26. - name: nginx
    27.   newName: my.image.registry/nginx
    28.   newTag: 1.4.0
    29. EOF

    Run kubectl kustomize ./ to see that the image being used is updated:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: my-nginx
    5. spec:
    6.   replicas: 2
    7.   selector:
    8.     matchLabels:
    9.       run: my-nginx
    10.   template:
    11.     metadata:
    12.       labels:
    13.         run: my-nginx
    14.     spec:
    15.       containers:
    16.       - image: my.image.registry/nginx:1.4.0
    17.         name: my-nginx
    18.         ports:
    19.         - containerPort: 80

    Sometimes, the application running in a Pod may need to use configuration values from other objects. For example, a Pod from a Deployment object need to read the corresponding Service name from Env or as a command argument. Since the Service name may change as namePrefix or nameSuffix is added in the kustomization.yaml file. It is not recommended to hard code the Service name in the command argument. For this usage, Kustomize can inject the Service name into containers through vars.

    1. # Create a deployment.yaml file (quoting the here doc delimiter)
    2. cat <<'EOF' > deployment.yaml
    3. apiVersion: apps/v1
    4. kind: Deployment
    5. metadata:
    6.   name: my-nginx
    7. spec:
    8.   selector:
    9.     matchLabels:
    10.       run: my-nginx
    11.   replicas: 2
    12.   template:
    13.     metadata:
    14.       labels:
    15.         run: my-nginx
    16.     spec:
    17.       containers:
    18.       - name: my-nginx
    19.         image: nginx
    20.         command: ["start", "--host", "$(MY_SERVICE_NAME)"]
    21. EOF
    23. # Create a service.yaml file
    24. cat <<EOF > service.yaml
    25. apiVersion: v1
    26. kind: Service
    27. metadata:
    28.   name: my-nginx
    29.   labels:
    30.     run: my-nginx
    31. spec:
    32.   ports:
    33.   - port: 80
    34.     protocol: TCP
    35.   selector:
    36.     run: my-nginx
    37. EOF
    39. cat <<EOF >./kustomization.yaml
    40. namePrefix: dev-
    41. nameSuffix: "-001"
    43. resources:
    44. - deployment.yaml
    45. - service.yaml
    47. vars:
    48. - name: MY_SERVICE_NAME
    49.   objref:
    50.     kind: Service
    51.     name: my-nginx
    52.     apiVersion: v1
    53. EOF

    Run kubectl kustomize ./ to see that the Service name injected into containers is dev-my-nginx-001:

    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4.   name: dev-my-nginx-001
    5. spec:
    6.   replicas: 2
    7.   selector:
    8.     matchLabels:
    9.       run: my-nginx
    10.   template:
    11.     metadata:
    12.       labels:
    13.         run: my-nginx
    14.     spec:
    15.       containers:
    16.       - command:
    17.         - start
    18.         - --host
    19.         - dev-my-nginx-001
    20.         image: nginx
    21.         name: my-nginx

    Kustomize has the concepts of bases and overlays. A base is a directory with a kustomization.yaml, which contains a set of resources and associated customization. A base could be either a local directory or a directory from a remote repo, as long as a kustomization.yaml is present inside. An overlay is a directory with a kustomization.yaml that refers to other kustomization directories as its bases. A base has no knowledge of an overlay and can be used in multiple overlays. An overlay may have multiple bases and it composes all resources from bases and may also have customization on top of them.

    Here is an example of a base:

    1. # Create a directory to hold the base
    2. mkdir base
    3. # Create a base/deployment.yaml
    4. cat <<EOF > base/deployment.yaml
    5. apiVersion: apps/v1
    6. kind: Deployment
    7. metadata:
    8.   name: my-nginx
    9. spec:
    10.   selector:
    11.     matchLabels:
    12.       run: my-nginx
    13.   replicas: 2
    14.   template:
    15.     metadata:
    16.       labels:
    17.         run: my-nginx
    18.     spec:
    19.       containers:
    20.       - name: my-nginx
    21.         image: nginx
    22. EOF
    24. # Create a base/service.yaml file
    25. cat <<EOF > base/service.yaml
    26. apiVersion: v1
    27. kind: Service
    28. metadata:
    29.   name: my-nginx
    30.   labels:
    31.     run: my-nginx
    32. spec:
    33.   ports:
    34.   - port: 80
    35.     protocol: TCP
    36.   selector:
    37.     run: my-nginx
    38. EOF
    39. # Create a base/kustomization.yaml
    40. cat <<EOF > base/kustomization.yaml
    41. resources:
    42. - deployment.yaml
    43. - service.yaml
    44. EOF

    This base can be used in multiple overlays. You can add different namePrefix or other cross-cutting fields in different overlays. Here are two overlays using the same base.

    1. mkdir dev
    2. cat <<EOF > dev/kustomization.yaml
    3. bases:
    4. - ../base
    5. namePrefix: dev-
    6. EOF
    8. mkdir prod
    9. cat <<EOF > prod/kustomization.yaml
    10. bases:
    11. - ../base
    12. namePrefix: prod-
    13. EOF

    How to apply/view/delete objects using Kustomize

    Use --kustomize or -k in kubectl commands to recognize Resources managed by kustomization.yaml. Note that -k should point to a kustomization directory, such as

    1. kubectl apply -k <kustomization directory>/

    Given the following kustomization.yaml,

    1. # Create a deployment.yaml file
    2. cat <<EOF > deployment.yaml
    3. apiVersion: apps/v1
    4. kind: Deployment
    5. metadata:
    6.   name: my-nginx
    7. spec:
    8.   selector:
    9.     matchLabels:
    10.       run: my-nginx
    11.   replicas: 2
    12.   template:
    13.     metadata:
    14.       labels:
    15.         run: my-nginx
    16.     spec:
    17.       containers:
    18.       - name: my-nginx
    19.         image: nginx
    20.         ports:
    21.         - containerPort: 80
    22. EOF
    24. # Create a kustomization.yaml
    25. cat <<EOF >./kustomization.yaml
    26. namePrefix: dev-
    27. commonLabels:
    28.   app: my-nginx
    29. resources:
    30. - deployment.yaml
    31. EOF

    Run the following command to apply the Deployment object dev-my-nginx:

    1. > kubectl apply -k ./
    2. deployment.apps/dev-my-nginx created

    Run one of the following commands to view the Deployment object dev-my-nginx:

    1. kubectl get -k ./
    1. kubectl describe -k ./

    Run the following command to compare the Deployment object dev-my-nginx against the state that the cluster would be in if the manifest was applied:

    1. > kubectl delete -k ./

    What’s next