Configure Access to Multiple Clusters

    Note: A file that is used to configure access to a cluster is sometimes called a kubeconfig file. This is a generic way of referring to configuration files. It does not mean that there is a file named kubeconfig.

    Warning: Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure. If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script.

    You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using or you can use one of these Kubernetes playgrounds:

    To check that kubectl is installed, run kubectl version --client. The kubectl version should be of your cluster’s API server.

    Define clusters, users, and contexts

    Suppose you have two clusters, one for development work and one for scratch work. In the development cluster, your frontend developers work in a namespace called frontend, and your storage developers work in a namespace called storage. In your scratch cluster, developers work in the default namespace, or they create auxiliary namespaces as they see fit. Access to the development cluster requires authentication by certificate. Access to the scratch cluster requires authentication by username and password.

    Create a directory named config-exercise. In your config-exercise directory, create a file named config-demo with this content:

    A configuration file describes clusters, users, and contexts. Your config-demo file has the framework to describe two clusters, two users, and three contexts.

    Go to your config-exercise directory. Enter these commands to add cluster details to your configuration file:

    1. kubectl config --kubeconfig=config-demo set-cluster development --server=https://1.2.3.4 --certificate-authority=fake-ca-file
    2. kubectl config --kubeconfig=config-demo set-cluster scratch --server=https://5.6.7.8 --insecure-skip-tls-verify

    Add user details to your configuration file:

    Caution: Storing passwords in Kubernetes client config is risky. A better alternative would be to use a credential plugin and store them separately. See: 

    1. kubectl config --kubeconfig=config-demo set-credentials developer --client-certificate=fake-cert-file --client-key=fake-key-seefile
    2. kubectl config --kubeconfig=config-demo set-credentials experimenter --username=exp --password=some-password

    Note:

    • To delete a user you can run kubectl --kubeconfig=config-demo config unset users.<name>
    • To remove a cluster, you can run kubectl --kubeconfig=config-demo config unset clusters.<name>
    • To remove a context, you can run kubectl --kubeconfig=config-demo config unset contexts.<name>

    Add context details to your configuration file:

    1. kubectl config --kubeconfig=config-demo set-context dev-frontend --cluster=development --namespace=frontend --user=developer
    2. kubectl config --kubeconfig=config-demo set-context dev-storage --cluster=development --namespace=storage --user=developer
    3. kubectl config --kubeconfig=config-demo set-context exp-scratch --cluster=scratch --namespace=default --user=experimenter

    Open your config-demo file to see the added details. As an alternative to opening the config-demo file, you can use the config view command.

    1. kubectl config --kubeconfig=config-demo view

    The output shows the two clusters, two users, and three contexts:

    1. apiVersion: v1
    2. clusters:
    3. - cluster:
    4.     certificate-authority: fake-ca-file
    5.     server: https://1.2.3.4
    6.   name: development
    7. - cluster:
    8.     insecure-skip-tls-verify: true
    9.     server: https://5.6.7.8
    10. contexts:
    11. - context:
    12.     cluster: development
    13.     namespace: frontend
    14.     user: developer
    15.   name: dev-frontend
    16. - context:
    17.     cluster: development
    18.     namespace: storage
    19.     user: developer
    20.   name: dev-storage
    21. - context:
    22.     cluster: scratch
    23.     namespace: default
    24.     user: experimenter
    25.   name: exp-scratch
    26. current-context: ""
    27. kind: Config
    29. users:
    30. - name: developer
    31.   user:
    32.     client-certificate: fake-cert-file
    33.     client-key: fake-key-file
    34. - name: experimenter
    35.   user:
    36.     # Documentation note (this comment is NOT part of the command output).
    37.     # Storing passwords in Kubernetes client config is risky.
    38.     # A better alternative would be to use a credential plugin
    39.     # and store the credentials separately.
    40.     # See https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
    41.     password: some-password
    42.     username: exp

    Sometimes you may want to use Base64-encoded data embedded here instead of separate certificate files; in that case you need to add the suffix -data to the keys, for example, certificate-authority-data, client-certificate-data, client-key-data.

    Each context is a triple (cluster, user, namespace). For example, the dev-frontend context says, “Use the credentials of the developer user to access the frontend namespace of the development cluster”.

    Set the current context:

    1. kubectl config --kubeconfig=config-demo use-context dev-frontend

    Now whenever you enter a kubectl command, the action will apply to the cluster, and namespace listed in the dev-frontend context. And the command will use the credentials of the user listed in the dev-frontend context.

    To see only the configuration information associated with the current context, use the --minify flag.

    1. kubectl config --kubeconfig=config-demo view --minify

    The output shows configuration information associated with the dev-frontend context:

    Now suppose you want to work for a while in the scratch cluster.

    Change the current context to exp-scratch:

    1. kubectl config --kubeconfig=config-demo use-context exp-scratch

    Now any kubectl command you give will apply to the default namespace of the scratch cluster. And the command will use the credentials of the user listed in the exp-scratch context.

    View configuration associated with the new current context, exp-scratch.

    1. kubectl config --kubeconfig=config-demo view --minify

    Finally, suppose you want to work for a while in the storage namespace of the development cluster.

    Change the current context to dev-storage:

    1. kubectl config --kubeconfig=config-demo use-context dev-storage

    View configuration associated with the new current context, dev-storage.

    1. kubectl config --kubeconfig=config-demo view --minify

    Create a second configuration file

    In your config-exercise directory, create a file named config-demo-2 with this content:

    1. kind: Config
    2. preferences: {}
    4. contexts:
    5. - context:
    7.     namespace: ramp
    8.     user: developer
    9.   name: dev-ramp-up

    See whether you have an environment variable named KUBECONFIG. If so, save the current value of your KUBECONFIG environment variable, so you can restore it later. For example:

    1. export KUBECONFIG_SAVED="$KUBECONFIG"

    Windows PowerShell

    1. $Env:KUBECONFIG_SAVED=$ENV:KUBECONFIG

    The KUBECONFIG environment variable is a list of paths to configuration files. The list is colon-delimited for Linux and Mac, and semicolon-delimited for Windows. If you have a KUBECONFIG environment variable, familiarize yourself with the configuration files in the list.

    Temporarily append two paths to your KUBECONFIG environment variable. For example:

    Windows PowerShell

    1. $Env:KUBECONFIG=("config-demo;config-demo-2")

    In your config-exercise directory, enter this command:

    1. kubectl config view

    The output shows merged information from all the files listed in your KUBECONFIG environment variable. In particular, notice that the merged information has the dev-ramp-up context from the config-demo-2 file and the three contexts from the config-demo file:

    1. contexts:
    2. - context:
    3.     cluster: development
    4.     namespace: frontend
    5.     user: developer
    6.   name: dev-frontend
    7. - context:
    8.     cluster: development
    9.     namespace: ramp
    10.     user: developer
    11.   name: dev-ramp-up
    12. - context:
    13.     cluster: development
    14.     namespace: storage
    15.     user: developer
    16.   name: dev-storage
    17. - context:
    18.     cluster: scratch
    19.     namespace: default
    20.     user: experimenter
    21.   name: exp-scratch

    For more information about how kubeconfig files are merged, see Organizing Cluster Access Using kubeconfig Files

    Explore the $HOME/.kube directory

    If you already have a cluster, and you can use kubectl to interact with the cluster, then you probably have a file named config in the $HOME/.kube directory.

    Go to $HOME/.kube, and see what files are there. Typically, there is a file named config. There might also be other configuration files in this directory. Briefly familiarize yourself with the contents of these files.

    Append $HOME/.kube/config to your KUBECONFIG environment variable

    If you have a $HOME/.kube/config file, and it’s not already listed in your KUBECONFIG environment variable, append it to your KUBECONFIG environment variable now. For example:

    1. export KUBECONFIG="${KUBECONFIG}:${HOME}/.kube/config"

    Windows Powershell

    1. $Env:KUBECONFIG="$Env:KUBECONFIG;$HOME\.kube\config"

    View configuration information merged from all the files that are now listed in your KUBECONFIG environment variable. In your config-exercise directory, enter:

    1. kubectl config view

    Return your KUBECONFIG environment variable to its original value. For example: 

    1. export KUBECONFIG="$KUBECONFIG_SAVED"

    Windows PowerShell

    Check the subject represented by the kubeconfig

    It is not always obvious what attributes (username, groups) you will get after authenticating to the cluster. It can be even more challenging if you are managing more than one cluster at the same time.

    There is a alpha subcommand command to check subject attributes, such as username, for your selected Kubernetes client context: kubectl alpha auth whoami.

    Read to learn about this in more detail.

    What’s next