Antrea Multi-cluster Quick Start

    The diagram below shows the two clusters and the ClusterSet to be created (for simplicity, the diagram just shows two Nodes for each cluster).

    We assume an Antrea version >= v1.8.0 is used in this guide, and the Antrea version is set to an environment variable TAG. For example, the following command sets the Antrea version to v1.8.0.

    To use the latest version of Antrea Multi-cluster from the Antrea main branch, you can change the YAML manifest path to: https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/ when applying or downloading an Antrea YAML manifest.

    Antrea must be deployed in both cluster A and cluster B, and the Multicluster feature of antrea-agent must be enabled to support multi-cluster Services. As we use ClusterIP endpoint type for multi-cluster Services, an Antrea Multi-cluster Gateway needs be set up in each member cluster to route Service traffic across clusters, and two clusters must have non-overlapping Service CIDRs. Set the following configuration parameters in antrea-agent.conf of the Antrea deployment manifest to enable the Multicluster feature:

    1. antrea-agent.conf: |
    2. ...
    3.   featureGates:
    4. ...
    5. ...
    6.   multicluster:
    7.     enableGateway: true
    8.     namespace: ""

    At the moment, Multi-cluster Gateway only works with the Antrea encap traffic mode, and all member clusters in a ClusterSet must use the same tunnel type.

    antctl provides a couple of commands to facilitate deployment, configuration, and troubleshooting of Antrea Multi-cluster. This section describes the steps to deploy Antrea Multi-cluster and set up the example ClusterSet using antctl. A further section will describe the steps to achieve the same using YAML manifests.

    To execute any command in this section, antctl needs access to the target cluster’s API server, and it needs a kubeconfig file for that. Please refer to the to learn more about the kubeconfig file configuration, and the antctl Multi-cluster commands. For installation of antctl, please refer to the installation guide.

    Step 1 - deploy Antrea Multi-cluster Controllers for leader and member

    Run the following commands to deploy Multi-cluster Controller for the leader into Namespace antrea-multicluster (Namespace antrea-multicluster will be created by the commands), and Multi-cluster Controller for the member into Namepsace kube-system.

    1. $kubectl create ns antrea-multicluster
    2. $antctl mc deploy leadercluster -n antrea-multicluster --antrea-version $TAG
    3. $antctl mc deploy membercluster -n kube-system --antrea-version $TAG

    You can run the following command to verify the the leader and member antrea-mc-controller Pods are deployed and running:

    1. $kubectl get all -A -l="component=antrea-mc-controller"
    2. NAMESPACE             NAME                                        READY   STATUS    RESTARTS   AGE
    3. antrea-multicluster   pod/antrea-mc-controller-cd7bf8f68-kh4kz    1/1     Running   0          50s
    4. kube-system           pod/antrea-mc-controller-85dbf58b75-pjj48   1/1     Running   0          48s
    7. antrea-multicluster   deployment.apps/antrea-mc-controller   1/1     1            1           50s
    8. kube-system           deployment.apps/antrea-mc-controller   1/1     1            1           48s

    Step 2 - initialize ClusterSet

    Run the following commands to create a ClusterSet with cluster A to be the leader, and also join the ClusterSet as a member.

    1. antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader -n antrea-multicluster --create-token -j join-config.yml
    2. antctl mc join --clusterid test-cluster-leader -n kube-system --config-file join-config.yml
    1. antctl mc create membertoken test-cluster-leader-token -n antrea-multicluster -o test-cluster-leader-token.yml
    2. antctl mc join --clusterid test-cluster-leader -n kube-system --config-file join-config.yml --token-secret-file test-cluster-leader-token.yml

    Step 3 - specify Multi-cluster Gateway Node

    Last, you need to choose at least one Node in cluster A to serve as the Multi-cluster Gateway. The Node should have an IP that is reachable from the cluster B’s Gateway Node, so a tunnel can be created between the two Gateways. For more information about Multi-cluster Gateway, please refer to the .

    Assuming K8s Node node-a1 is selected for the Multi-cluster Gateway, run the following command to annotate the Node with: multicluster.antrea.io/gateway=true (so Antrea can know it is the Gateway Node from the annotation):

    1. $kubectl annotate node node-a1 multicluster.antrea.io/gateway=true

    Let us switch to cluster B. All the kubectl and antctl commands in the following steps should be run with the kubeconfig for cluster B.

    Step 1 - deploy Antrea Multi-cluster Controller for member

    Run the following command to deploy the member Multi-cluster Controller into Namespace kube-system.

    You can run the following command to verify the antrea-mc-controller Pod is deployed and running:

    1. $kubectl get all -A -l="component=antrea-mc-controller"
    2. NAMESPACE             NAME                                        READY   STATUS    RESTARTS   AGE
    3. kube-system           pod/antrea-mc-controller-85dbf58b75-pjj48   1/1     Running   0          40s
    5. NAMESPACE             NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE

    Step 2 - join ClusterSet

    Run the following command to make cluster B join the ClusterSet:

    1. antctl mc join --clusterid test-cluster-member -n kube-system --config-file join-config.yml

    join-config.yml is generated when creating the ClusterSet in cluster A. Again, you can also run the antctl mc create membertoken in the leader cluster (cluster A) to create a separate token for cluster B, and join using that token, rather than the default token in join-config.yml.

    Step 3 - specify Multi-cluster Gateway Node

    Assuming K8s Node node-b1 is chosen to be the Multi-cluster Gateway for cluster B, run the following command to annotate the Node:

    1. $kubectl annotate node node-b1 multicluster.antrea.io/gateway=true

    So far, we set up an Antrea Multi-cluster ClusterSet with two clusters following the above sections of this guide. Next, you can start to consume the Antrea Multi-cluster features with the ClusterSet, including , Multi-cluster NetworkPolicy, and , Please check the relevant Antrea Multi-cluster User Guide sections to learn more.

    If you want to add a new member cluster to your ClusterSet, you can follow the steps for cluster B to do so. For example, you can run the following command to join the ClusterSet in a member cluster with ID test-cluster-member2:

    1. antctl mc join --clusterid test-cluster-member2 -n kube-system --config-file join-config.yml

    Step 1 - deploy Antrea Multi-cluster Controllers for leader and member

    Run the following commands to deploy Multi-cluster Controller for the leader into Namespace antrea-multicluster (Namespace antrea-multicluster will be created by the commands), and Multi-cluster Controller for the member into Namepsace .

    1. $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml
    2. $kubectl create ns antrea-multicluster
    3. $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml
    4. $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml

    Step 2 - initialize ClusterSet

    1. $kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/leader-clusterset-template.yml
    2. $kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/leader-access-token-template.yml
    3. $kubectl get secret default-member-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e '  ca.crt' -e '  token:' -e '^type' -e '  namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > default-member-token.yml

    The last command saves the token Secret manifest to default-member-token.yml, which will be needed for member clusters to join the ClusterSet. Note, in this example, we use a shared token for all member clusters. If you want to use a separate token for each member cluster for security considerations, you can follow the instructions in the Multi-cluster User Guide.

    Next, run the following commands to make cluster A join the ClusterSet also as a member:

    Here, 172.10.0.11 is the kube-apiserver IP of cluster A. You should replace it with the kube-apiserver IP of your leader cluster.

    Step 3 - specify Multi-cluster Gateway Node

    Assuming K8s Node node-a1 is selected for the Multi-cluster Gateway, run the following command to annotate the Node:

    1. $kubectl annotate node node-a1 multicluster.antrea.io/gateway=true

    Let us switch to cluster B. All the kubectl commands in the following steps should be run with the kubeconfig for cluster B.

    Step 1 - deploy Antrea Multi-cluster Controller for member

    Run the following command to deploy the member Multi-cluster Controller into Namespace kube-system.

    1. $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml

    You can run the following command to verify the antrea-mc-controller Pod is deployed and running:

    1. $kubectl get all -A -l="component=antrea-mc-controller"
    2. NAMESPACE             NAME                                        READY   STATUS    RESTARTS   AGE
    3. kube-system           pod/antrea-mc-controller-85dbf58b75-pjj48   1/1     Running   0          40s
    5. NAMESPACE             NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
    6. kube-system           deployment.apps/antrea-mc-controller   1/1     1            1           40s

    Step 2 - join ClusterSet

    Run the following commands to make cluster B join the ClusterSet:

    1. $kubectl apply -f default-member-token.yml
    2. $curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/member-clusterset-template.yml > member-clusterset.yml
    3. $sed -e 's/<LEADER_APISERVER_IP>/172.10.0.11/g' member-clusterset.yml | kubectl apply -f -

    default-member-token.yml saves the default member token which was generated when initializing the ClusterSet in cluster A.

    Step 3 - specify Multi-cluster Gateway Node

    Assuming K8s Node node-b1 is chosen to be the Multi-cluster Gateway for cluster B, run the following command to annotate the Node:

    1. $kubectl annotate node node-b1 multicluster.antrea.io/gateway=true

    If you want to add a new member cluster to your ClusterSet, you can follow the steps for cluster B to do so. Remember to update the member cluster ID in member-clusterset-template.yml to the new member cluster’s ID in the step 2 of joining ClusterSet. For example, you can run the following commands to join the ClusterSet in a member cluster with ID test-cluster-member2:

    1. $kubectl apply -f default-member-token.yml
    2. $curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/member-clusterset-template.yml  > member-clusterset.yml
    3. $sed -e 's/<LEADER_APISERVER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' member-clusterset.yml | kubectl apply -f -