Using Antrea with Multus

    Multus, in order to attach multiple network interfaces to Pods. In this scenario, Antrea is used for the default network, i.e. it is the CNI plugin which provisions the “primary” network interface (“eth0”) for each Pod. For the sake of this guide, we will use the

    CNI plugin to provision secondary network interfaces for selected Pods, but similar steps should apply for other plugins as well, e.g. ipvlan.

    The general prerequisites are:

    • a K8s cluster (Linux Nodes) running a K8s version supported by Antrea. At the time of writing, we recommend version 1.16 or later. Typically the cluster needs to be running on a network infrastructure that you control. For example, using macvlan networking will not work on public clouds like AWS.
    • kubectl

    The Antrea IPAM capability for secondary network was added in Antrea version 1.7. To leverage Antrea IPAM for IP assignment of secondary networks, an Antrea version >= 1.7.0 should be used. There is no Antrea version requirement for other IPAM options.

    All the required software will be deployed using YAML manifests, and the corresponding container images will be downloaded from public registries.

    For the sake of this guide, we will use macvlan in “bridge” mode, which supports the creation of multiple subinterfaces on one parent interface, and connects them all using a bridge. Macvlan in “bridge” mode requires the network to be able to handle “promiscuous mode”, as the same physical interface / virtual adapter ends up being assigned multiple MAC addresses. When using a virtual network for the Nodes, some configuration changes are usually required, which depend on the virtualization technology. For example:

    • when using VirtualBox and , set the Promiscuous Mode to Allow All
    • when using VMware Fusion, enable “promiscuous mode” in the guest (Node) for the appropriate interface (e.g. using ifconfig); this may prompt for your password on the host unless you uncheck Require authentication to enter promiscuous mode in Preferences ... > Network

    This needs to be done for every Node VM, so it’s best if you can automate this when provisioning your VMs.

    If you need to create a K8s cluster to test this guide, we suggest you create one by following . You will need to use a slightly modified Vagrantfile, which you can find

    here. Note that this Vagrantfile will create 3 VMs on your machine, and each VM will be allocated 2GB of memory, so make sure you have enough memory available. You can create the cluster with the following steps:

    The last command will take around 10 to 15 minutes to complete. After that, your cluster is ready and you can set the KUBECONFIG environment variable in order to use kubectl:

    1. export KUBECONFIG=`pwd`/infra/vagrant/playbook/kube/config
    2. kubectl cluster-info

    The cluster that you have created by following these steps is the one we will use as an example in this guide.

    Practical steps

    Step 1: Deploying Antrea

    . You may also choose a released Antrea version.

    To leverage Antrea IPAM to assign IP addresses for the secondary network, you need to edit the Antrea deployment manifest and enable the AntreaIPAM feature gate for both antrea-controller and antrea-agent, and then deploy Antrea with:

    1. kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea.yml

    If you choose other IPAM options like DHCP or Whereabouts, you can just deploy Antrea with the Antrea deployment manifest without modification:

    1. kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea.yml
    1. git clone https://github.com/k8snetworkplumbingwg/multus-cni && cd multus-cni
    2. cat ./deployments/multus-daemonset-thick-plugin.yml | kubectl apply -f -

    Step 3: Create an IPPool and a NetworkAttachmentDefinition

    With Antrea IPAM, the subnet and IP ranges for the secondary network are defined with an Antrea IPPool CR. To learn more information about Antrea IPAM for secondary network, please refer to the .

    Next, you should create a NetworkAttachmentDefinition for the macvlan network. Make sure that you are using the correct interface name for "master": it should be the name of the Node’s network interface (e.g. eth0) that you want to use as the parent interface for the macvlan secondary Pod interfaces. If you are using our test cluster, enp0s9 is the correct value.

    1. cat <<EOF | kubectl create -f -
    2. apiVersion: "k8s.cni.cncf.io/v1"
    3. kind: NetworkAttachmentDefinition
    4. metadata:
    5.   name: macvlan-conf
    6. spec:
    7.   config: '{
    8.       "cniVersion": "0.3.0",
    9.       "type": "macvlan",
    10.       "master": "enp0s9",
    11.       "mode": "bridge",
    12.       "ipam": {
    13.         "type": "antrea",
    14.         "ippools": [ "macvlan-ippool" ]
    15.       }
    16. EOF

    Again, the above definition assumes that Antrea IPAM will be used to assign IP addresses to the macvlan secondary interfaces. Section and section Using Whereabouts for IPAM talk about alternative IPAM solutions with DHCP and the project respectively.

    This step is required if you want the Node to be able to communicate with the Pods using the secondary network:

    2. # edit file as needed
    3. kubectl apply -f macvlan-host-init.yml

    If you are using our , no edits are required and you can apply the file as is. Otherwise, make sure that you replace enp0s9 with the name of the parent network interface.

    This manifest will create a DaemonSet that will run a bash script once on every Node. It will:

    • Enable promiscuous mode on the parent interface using ifconfig; if using a virtual network for the Nodes, this needs to be done in addition to enabling promiscuous mode in the hypervisor for the virtual adapters, as described in the Prerequisites.
    • Create a macvlan subinterface for the Node
    • Move the IP address from the parent interface to the new subinterface

    No edit to the manifest should be required, regardless of which K8s cluster you are using.

    To use DHCP for IPAM, you need not create the Antrea IPPool CR in step-3, and dhcp should be specified as the IPAM type in the NetworkAttachmentDefinition CR created. And, two more extra steps are required, after step 1 - 4.

    Step 5: Run a DHCP server

    Note that in order to deploy the DHCP server in-cluster, step-4 is required. You can then apply the following manifest after making the necessary edits:

    1. wget https://raw.githubusercontent.com/antrea-io/antrea/main/docs/cookbooks/multus/resources/dhcp-server.yml
    2. # edit file as needed
    3. kubectl apply -f dhcp-server.yml

    Once again, if you are using our , no edits are required and you can apply the file as is. Otherwise, make sure that you edit the dhcp-server-conf ConfigMap at the top of the file: the

    dhcpd.conf configuration data needs to be correct for our use case. You will need to ensure that the subnet matches the parent interface’s network and that the range of allocatable addresses does not include the IP addresses assigned to the parent interfaces.

    When using the dhcp plugin to assign IP addresses to Pod interfaces, a separate daemon needs to run on each Node, which will notably ensure that local DHCP leases are renewed periodically. To deploy the DHCP daemon (as a DaemonSet), you can use the following command:

    1. kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/docs/cookbooks/multus/resources/dhcp-daemon.yml

    Testing

    To test that the secondary interfaces are functional, you can create Pods with the k8s.v1.cni.cncf.io/networks: macvlan-conf annotation. We suggest creating a set of Pods (as a Deployment) which all request a macvlan secondary interface, using the provided manifest:

    You can then kubectl exec into the Pods to inspect the networking configuration and have Pods ping each other using the secondary network.

    1. $ kubectl get pods -o wide
    2. NAME                        READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
    3. samplepod-7956c4498-65v6m   1/1     Running   0          68s   10.10.2.10   k8s-node-worker-2   <none>           <none>
    4. samplepod-7956c4498-9dz98   1/1     Running   0          68s   10.10.1.12   k8s-node-worker-1   <none>           <none>
    5. samplepod-7956c4498-ghrdg   1/1     Running   0          68s   10.10.1.13   k8s-node-worker-1   <none>           <none>
    6. samplepod-7956c4498-n65bn   1/1     Running   0          68s   10.10.2.12   k8s-node-worker-2   <none>           <none>
    7. samplepod-7956c4498-q6vp2   1/1     Running   0          68s   10.10.1.11   k8s-node-worker-1   <none>           <none>
    8. samplepod-7956c4498-xztf4   1/1     Running   0          68s   10.10.2.11   k8s-node-worker-2   <none>           <none>
    1. $ kubectl exec samplepod-7956c4498-65v6m -- ip addr
    2. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    3.     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    4.     inet 127.0.0.1/8 scope host lo
    5.        valid_lft forever preferred_lft forever
    6. 3: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    7.     link/ether c2:ce:36:6b:ba:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    8.     inet 10.10.2.10/24 brd 10.10.2.255 scope global eth0
    9.        valid_lft forever preferred_lft forever
    10. 4: net1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    11.     link/ether be:a0:35:f2:08:2d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    12.     inet 192.168.78.205/24 brd 192.168.78.255 scope global net1
    13.        valid_lft forever preferred_lft forever
    1. $ kubectl exec samplepod-7956c4498-9dz98 -- ip addr
    2. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    3.     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    4.     inet 127.0.0.1/8 scope host lo
    5.        valid_lft forever preferred_lft forever
    6. 3: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    7.     inet 10.10.1.12/24 brd 10.10.1.255 scope global eth0
    8.        valid_lft forever preferred_lft forever
    9. 4: net1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    10.     link/ether 22:6e:b1:0a:f3:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
    11.     inet 192.168.78.202/24 brd 192.168.78.255 scope global net1
    12.        valid_lft forever preferred_lft forever
    1. $ kubectl exec samplepod-7956c4498-9dz98 -- ping -c 3 192.168.78.205
    2. PING 192.168.78.205 (192.168.78.205) 56(84) bytes of data.
    3. 64 bytes from 192.168.78.205: icmp_seq=1 ttl=64 time=0.846 ms
    4. 64 bytes from 192.168.78.205: icmp_seq=2 ttl=64 time=0.410 ms
    5. 64 bytes from 192.168.78.205: icmp_seq=3 ttl=64 time=0.507 ms

    The diagram below shows an overview of a K8s Node when using the using DHCP for IPAM, and following all the steps above. For the sake of completeness, we show the DHCP server running on that Node, but as we use a Deployment with a single replica, the server may be running on any worker Node in the cluster.

    Using Whereabouts for IPAM

    If you do not already have a DHCP server for the underlying parent network and you find that deploying one in-cluster is impractical, you may want to consider using to assign IP addresses to the secondary interfaces. When using Whereabouts, follow steps 1 and 2 above, along with step 4 if you want the Nodes to be able to communicate with the Pods using the secondary network.

    The next step is to install the plugin as follows:

    Then create a NetworkAttachmentDefinition like the one below, after ensuring that "master" matches the name of the parent interface on the Nodes, and that the range and exclude configuration parameters are correct for your cluster (in particular, make sure that you exclude IP addresses assigned to Nodes). If you are using our test cluster, you can use the NetworkAttachmentDefinition below as is.

    1. cat <<EOF | kubectl create -f -
    2. apiVersion: "k8s.cni.cncf.io/v1"
    3. kind: NetworkAttachmentDefinition
    4. metadata:
    5.   name: macvlan-conf
    6. spec:
    7.   config: '{
    8.       "cniVersion": "0.3.0",
    9.       "type": "macvlan",
    10.       "master": "enp0s9",
    11.       "mode": "bridge",
    12.       "ipam": {
    13.         "type": "whereabouts",
    14.         "datastore": "kubernetes",
    15.         "kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
    16.         "range": "192.168.78.0/24",
    17.         "exclude": [
    18.            "192.168.78.0/25",
    19.            "192.168.78.128/26",
    20.            "192.168.78.192/29",
    21.            "192.168.78.250/31",
    22.            "192.168.78.252/30"
    23.         ],
    24.         "log_file" : "/tmp/whereabouts.log",
    25.         "log_level" : "debug"
    26.       }
    27.     }'

    as above.