Using virtual functions (VFs) with DPDK and RDMA modes

    Prerequisites

    • Install the OpenShift CLI ().

    • Install the SR-IOV Network Operator.

    • Log in as a user with cluster-admin privileges.

    Procedure

    1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the intel-dpdk-node-policy.yaml file.

      1Specify the driver type for the virtual functions to vfio-pci.

      See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

      When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

      After the configuration update is applied, all the pods in openshift-sriov-network-operator namespace will change to a Running status.

    2. Create the SriovNetworkNodePolicy object by running the following command:

      1. $ oc create -f intel-dpdk-node-policy.yaml

    3. Create the following SriovNetwork object, and then save the YAML in the intel-dpdk-network.yaml file.

      1. apiVersion: sriovnetwork.openshift.io/v1
      2. kind: SriovNetwork
      3. metadata:
      4.   name: intel-dpdk-network
      5.   namespace: openshift-sriov-network-operator
      6. spec:
      7.   networkNamespace: <target_namespace>
      8.   ipam: "{}" (1)
      9.   vlan: <vlan>
      10.   resourceName: intelnics
      1Specify an empty object “{}” for the ipam CNI plug-in. DPDK works in userspace mode and does not require an IP address.

      See the “Configuring SR-IOV additional network” section for a detailed explanation on each option in SriovNetwork.

    4. Create the SriovNetwork object by running the following command:

      1. $ oc create -f intel-dpdk-network.yaml
      1. apiVersion: v1
      2. kind: Pod
      3. metadata:
      4.   name: dpdk-app
      5.   namespace: <target_namespace> (1)
      6.   annotations:
      7.     k8s.v1.cni.cncf.io/networks: intel-dpdk-network
      8. spec:
      9.   containers:
      10.   - name: testpmd
      11.     image: <DPDK_image> (2)
      12.     securityContext:
      13.       runAsUser: 0
      14.       capabilities:
      15.         add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] (3)
      16.     volumeMounts:
      17.     - mountPath: /dev/hugepages (4)
      18.       name: hugepage
      19.     resources:
      20.       limits:
      21.         openshift.io/intelnics: "1" (5)
      22.         memory: "1Gi"
      23.         cpu: "4" (6)
      24.         hugepages-1Gi: "4Gi" (7)
      25.       requests:
      26.         openshift.io/intelnics: "1"
      27.         memory: "1Gi"
      28.         cpu: "4"
      29.         hugepages-1Gi: "4Gi"
      30.     command: ["sleep", "infinity"]
      31.   volumes:
      32.   - name: hugepage
      33.     emptyDir:

    6. Create the DPDK pod by running the following command:

      1. $ oc create -f intel-dpdk-pod.yaml

    Prerequisites

    • Install the OpenShift CLI (oc).

    • Install the SR-IOV Network Operator.

    • Log in as a user with cluster-admin privileges.

    Procedure

    1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the mlx-dpdk-node-policy.yaml file.

      1Specify the device hex code of the SR-IOV network device. The only allowed values for Mellanox cards are 1015, 1017.
      2Specify the driver type for the virtual functions to netdevice. Mellanox SR-IOV VF can work in DPDK mode without using the vfio-pci device type. VF device appears as a kernel network interface inside a container.
      3Enable RDMA mode. This is required by Mellanox cards to work in DPDK mode.

      See the Configuring SR-IOV network devices section for detailed explanation on each option in SriovNetworkNodePolicy.

      When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

      After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

    2. Create the SriovNetworkNodePolicy object by running the following command:

      1. $ oc create -f mlx-dpdk-node-policy.yaml

    3. Create the following SriovNetwork object, and then save the YAML in the mlx-dpdk-network.yaml file.

      1. apiVersion: sriovnetwork.openshift.io/v1
      2. kind: SriovNetwork
      3. metadata:
      4.   name: mlx-dpdk-network
      5.   namespace: openshift-sriov-network-operator
      6. spec:
      7.   networkNamespace: <target_namespace>
      8.   ipam: |- (1)
      9.     ...
      10.   vlan: <vlan>
      11.   resourceName: mlxnics
      1Specify a configuration object for the ipam CNI plug-in as a YAML block scalar. The plug-in manages IP address assignment for the attachment definition.

      See the “Configuring SR-IOV additional network” section for a detailed explanation on each option in SriovNetwork.

    4. Create the SriovNetworkNodePolicy object by running the following command:

      1. $ oc create -f mlx-dpdk-network.yaml

    5. Create the following Pod spec, and then save the YAML in the mlx-dpdk-pod.yaml file.

      1. apiVersion: v1
      2. kind: Pod
      3. metadata:
      4.   name: dpdk-app
      5.   namespace: <target_namespace> (1)
      6.   annotations:
      7.     k8s.v1.cni.cncf.io/networks: mlx-dpdk-network
      8. spec:
      9.   containers:
      10.   - name: testpmd
      11.     image: <DPDK_image> (2)
      12.     securityContext:
      13.       runAsUser: 0
      14.       capabilities:
      15.         add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] (3)
      16.     volumeMounts:
      17.     - mountPath: /dev/hugepages (4)
      18.       name: hugepage
      19.     resources:
      20.       limits:
      21.         openshift.io/mlxnics: "1" (5)
      22.         memory: "1Gi"
      23.         cpu: "4" (6)
      24.         hugepages-1Gi: "4Gi" (7)
      25.       requests:
      26.         openshift.io/mlxnics: "1"
      27.         memory: "1Gi"
      28.         cpu: "4"
      29.         hugepages-1Gi: "4Gi"
      30.     command: ["sleep", "infinity"]
      31.   volumes:
      32.   - name: hugepage
      33.     emptyDir:
      1. $ oc create -f mlx-dpdk-pod.yaml

    RDMA over Converged Ethernet (RoCE) is the only supported mode when using RDMA on OKD.

    Prerequisites

    • Install the OpenShift CLI (oc).

    • Install the SR-IOV Network Operator.

    • Log in as a user with cluster-admin privileges.

    Procedure

    1. Create the following SriovNetworkNodePolicy object, and then save the YAML in the mlx-rdma-node-policy.yaml file.

      1Specify the device hex code of SR-IOV network device. The only allowed values for Mellanox cards are 1015, 1017.
      2Specify the driver type for the virtual functions to netdevice.
      3Enable RDMA mode.

      See the Configuring SR-IOV network devices section for a detailed explanation on each option in SriovNetworkNodePolicy.

      When applying the configuration specified in a SriovNetworkNodePolicy object, the SR-IOV Operator may drain the nodes, and in some cases, reboot nodes. It may take several minutes for a configuration change to apply. Ensure that there are enough available nodes in your cluster to handle the evicted workload beforehand.

      After the configuration update is applied, all the pods in the openshift-sriov-network-operator namespace will change to a Running status.

    2. Create the SriovNetworkNodePolicy object by running the following command:

      1. $ oc create -f mlx-rdma-node-policy.yaml

    3. Create the following SriovNetwork object, and then save the YAML in the mlx-rdma-network.yaml file.

      1. apiVersion: sriovnetwork.openshift.io/v1
      2. kind: SriovNetwork
      3. metadata:
      4.   name: mlx-rdma-network
      5.   namespace: openshift-sriov-network-operator
      6. spec:
      7.   networkNamespace: <target_namespace>
      8.   ipam: |- (1)
      9.     ...
      10.   vlan: <vlan>
      11.   resourceName: mlxnics
      1Specify a configuration object for the ipam CNI plug-in as a YAML block scalar. The plug-in manages IP address assignment for the attachment definition.

      See the “Configuring SR-IOV additional network” section for a detailed explanation on each option in SriovNetwork.

    4. Create the SriovNetworkNodePolicy object by running the following command:

      1. $ oc create -f mlx-rdma-network.yaml

    5. Create the following Pod spec, and then save the YAML in the mlx-rdma-pod.yaml file.

      1. apiVersion: v1
      2. kind: Pod
      3. metadata:
      4.   name: rdma-app
      5.   namespace: <target_namespace> (1)
      6.   annotations:
      7.     k8s.v1.cni.cncf.io/networks: mlx-rdma-network
      8. spec:
      9.   containers:
      10.   - name: testpmd
      11.     image: <RDMA_image> (2)
      12.     securityContext:
      13.       runAsUser: 0
      14.       capabilities:
      15.         add: ["IPC_LOCK","SYS_RESOURCE","NET_RAW"] (3)
      16.     volumeMounts:
      17.     - mountPath: /dev/hugepages (4)
      18.       name: hugepage
      19.     resources:
      20.       limits:
      21.         memory: "1Gi"
      22.         cpu: "4" (5)
      23.         hugepages-1Gi: "4Gi" (6)
      24.       requests:
      25.         memory: "1Gi"
      26.         cpu: "4"
      27.         hugepages-1Gi: "4Gi"
      28.     command: ["sleep", "infinity"]
      29.   volumes:
      30.   - name: hugepage
      31.     emptyDir:
      32.       medium: HugePages