Updating node network configuration

    OpenShift Container Platform uses nmstate to report on and configure the state of the node network. This makes it possible to modify network policy configuration, such as by creating a Linux bridge on all nodes, by applying a single configuration manifest to the cluster.

    Node networking is monitored and updated by the following objects:

    NodeNetworkState

    Reports the state of the network on that node.

    NodeNetworkConfigurationPolicy

    Describes the requested network configuration on nodes. You update the node network configuration, including adding and removing interfaces, by applying a NodeNetworkConfigurationPolicy manifest to the cluster.

    NodeNetworkConfigurationEnactment

    Reports the network policies enacted upon each node.

    OpenShift Container Platform supports the use of the following nmstate interface types:

    • Linux Bridge

    • VLAN

    • Bond

    • Ethernet

    Creating an interface on nodes

    Create an interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy manifest to the cluster. The manifest details the requested configuration for the interface.

    By default, the manifest applies to all nodes in the cluster. To add the interface to specific nodes, add the spec: nodeSelector parameter and the appropriate <key>:<value> for your node selector.

    Procedure

    1. Create the NodeNetworkConfigurationPolicy manifest. The following example configures a Linux bridge on all worker nodes:

      1Name of the policy.
      2Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.
      3This example uses the node-role.kubernetes.io/worker: “” node selector to select all worker nodes in the cluster.
      4Optional: Human-readable description for the interface.

    2. Create the node network policy:

      1. $ oc apply -f <br1-eth1-policy.yaml> (1)
      1File name of the node network configuration policy manifest.

    Confirming node network policy updates on nodes

    A NodeNetworkConfigurationPolicy manifest describes your requested network configuration for nodes in the cluster. The node network policy includes your requested network configuration and the status of execution of the policy on the cluster as a whole.

    When you apply a node network policy, a NodeNetworkConfigurationEnactment object is created for every node in the cluster. The node network configuration enactment is a read-only object that represents the status of execution of the policy on that node. If the policy fails to be applied on the node, the enactment for that node includes a traceback for troubleshooting.

    Procedure

    1. To confirm that a policy has been applied to the cluster, list the policies and their status:

      1. $ oc get nncp

    2. Optional: If a policy is taking longer than expected to successfully configure, you can inspect the requested state and status conditions of a particular policy:

      1. $ oc get nncp <policy> -o yaml

    3. Optional: If a policy is taking longer than expected to successfully configure on all nodes, you can list the status of the enactments on the cluster:

      1. $ oc get nnce

    4. Optional: To view the configuration of a particular enactment, including any error reporting for a failed configuration:

      1. $ oc get nnce <node>.<policy> -o yaml

    You can remove an interface from one or more nodes in the cluster by editing the NodeNetworkConfigurationPolicy object and setting the state of the interface to absent.

    Removing an interface from a node does not automatically restore the node network configuration to a previous state. If you want to restore the previous state, you will need to define that node network configuration in the policy.

    If you remove a bridge or bonding interface, any node NICs in the cluster that were previously attached or subordinate to that bridge or bonding interface are placed in a down state and become unreachable. To avoid losing connectivity, configure the node NIC in the same policy so that it has a status of up and either DHCP or a static IP address.

    Deleting the node network policy that added an interface does not change the configuration of the policy on the node. Although a NodeNetworkConfigurationPolicy is an object in the cluster, it only represents the requested configuration.
    Similarly, removing an interface does not delete the policy.

    Procedure

    1. Update the NodeNetworkConfigurationPolicy manifest used to create the interface. The following example removes a Linux bridge and configures the eth1 NIC with DHCP to avoid losing connectivity:

    2. Update the policy on the node and remove the interface:

      1. $ oc apply -f <br1-eth1-policy.yaml> (1)
      1File name of the policy manifest.

    Example policy configurations for different interfaces

    Create a Linux bridge interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy manifest to the cluster.

    The following YAML file is an example of a manifest for a Linux bridge interface. It includes samples values that you must replace with your own information.

    1. apiVersion: nmstate.io/v1beta1
    2. kind: NodeNetworkConfigurationPolicy
    3. metadata:
    4.   name: br1-eth1-policy (1)
    5. spec:
    6.   nodeSelector: (2)
    7.     kubernetes.io/hostname: <node01> (3)
    8.   desiredState:
    9.     interfaces:
    10.       - name: br1 (4)
    11.         description: Linux bridge with eth1 as a port (5)
    12.         type: linux-bridge (6)
    13.         state: up (7)
    14.         ipv4:
    15.           dhcp: true (8)
    16.           enabled: true (9)
    17.         bridge:
    18.           options:
    19.             stp:
    20.               enabled: false (10)
    21.             - name: eth1 (11)
    1Name of the policy.
    2Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.
    3This example uses a hostname node selector.
    4Name of the interface.
    5Optional: Human-readable description of the interface.
    6The type of interface. This example creates a bridge.
    7The requested state for the interface after creation.
    8Optional: If you do not use dhcp, you can either set a static IP or leave the interface without an IP address.
    9Enables ipv4 in this example.
    10Disables stp in this example.
    11The node NIC to which the bridge attaches.

    Example: VLAN interface node network configuration policy

    Create a VLAN interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy manifest to the cluster.

    The following YAML file is an example of a manifest for a VLAN interface. It includes samples values that you must replace with your own information.

    1. apiVersion: nmstate.io/v1beta1
    3. metadata:
    4.   name: vlan-eth1-policy (1)
    5. spec:
    6.   nodeSelector: (2)
    7.     kubernetes.io/hostname: <node01> (3)
    8.   desiredState:
    9.     interfaces:
    10.     - name: eth1.102 (4)
    11.       description: VLAN using eth1 (5)
    12.       type: vlan (6)
    13.       state: up (7)
    14.       vlan:
    15.         base-iface: eth1 (8)
    16.         id: 102 (9)
    1Name of the policy.
    2Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.
    3This example uses a hostname node selector.
    4Name of the interface.
    5Optional: Human-readable description of the interface.
    6The type of interface. This example creates a VLAN.
    7The requested state for the interface after creation.
    8The node NIC to which the VLAN is attached.
    9The VLAN tag.

    Example: Bond interface node network configuration policy

    The following YAML file is an example of a manifest for a bond interface. It includes samples values that you must replace with your own information.

    1. apiVersion: nmstate.io/v1beta1
    2. kind: NodeNetworkConfigurationPolicy
    3. metadata:
    4.   name: bond0-eth1-eth2-policy (1)
    5. spec:
    6.   nodeSelector: (2)
    7.     kubernetes.io/hostname: <node01> (3)
    8.   desiredState:
    9.     interfaces:
    10.     - name: bond0 (4)
    11.       description: Bond enslaving eth1 and eth2 (5)
    12.       type: bond (6)
    13.       state: up (7)
    14.       ipv4:
    15.         dhcp: true (8)
    16.         enabled: true (9)
    17.       link-aggregation:
    18.         mode: active-backup (10)
    19.         options:
    20.           miimon: '140' (11)
    21.         slaves: (12)
    22.         - eth1
    23.         - eth2
    24.       mtu: 1450 (13)
    1Name of the policy.
    2Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.
    3This example uses a hostname node selector.
    4Name of the interface.
    5Optional: Human-readable description of the interface.
    6The type of interface. This example creates a bond.
    7The requested state for the interface after creation.
    8Optional: If you do not use dhcp, you can either set a static IP or leave the interface without an IP address.
    9Enables ipv4 in this example.
    10The driver mode for the bond. This example uses an active backup mode.
    11Optional: This example uses miimon to inspect the bond link every 140ms.
    12The subordinate node NICs in the bond.
    13Optional: The maximum transmission unit (MTU) for the bond. If not specified, this value is set to 1500 by default.

    Configure an Ethernet interface on nodes in the cluster by applying a NodeNetworkConfigurationPolicy manifest to the cluster.

    The following YAML file is an example of a manifest for an Ethernet interface. It includes sample values that you must replace with your own information.

    1. apiVersion: nmstate.io/v1beta1
    2. kind: NodeNetworkConfigurationPolicy
    3. metadata:
    4.   name: eth1-policy (1)
    5. spec:
    6.   nodeSelector: (2)
    7.     kubernetes.io/hostname: <node01> (3)
    8.   desiredState:
    9.     interfaces:
    10.     - name: eth1 (4)
    11.       description: Configuring eth1 on node01 (5)
    12.       state: up (7)
    13.       ipv4:
    14.         dhcp: true (8)
    1Name of the policy.
    2Optional: If you do not include the nodeSelector parameter, the policy applies to all nodes in the cluster.
    3This example uses a hostname node selector.
    4Name of the interface.
    5Optional: Human-readable description of the interface.
    6The type of interface. This example creates an Ethernet networking interface.
    7The requested state for the interface after creation.
    8Optional: If you do not use dhcp, you can either set a static IP or leave the interface without an IP address.
    9Enables ipv4 in this example.

    Example: Multiple interfaces in the same node network configuration policy

    You can create multiple interfaces in the same node network configuration policy. These interfaces can reference each other, allowing you to build and deploy a network configuration by using a single policy manifest.

    The following example snippet creates a bond that is named bond10 across two NICs and a Linux bridge that is named br1 that connects to the bond.

    The following example configuration snippets demonstrate different methods of IP management.

    These examples use the ethernet interface type to simplify the example while showing the related context in the policy configuration. These IP management examples can be used with the other interface types.

    Static

    The following snippet statically configures an IP address on the Ethernet interface:

    1. ...
    2.     interfaces:
    3.     - name: eth1
    4.       description: static IP on eth1
    5.       type: ethernet
    6.       state: up
    7.       ipv4:
    8.         address:
    9.         - ip: 192.168.122.250 (1)
    10.           prefix-length: 24
    11.         enabled: true
    12. ...
    1Replace this value with the static IP address for the interface.

    The following snippet ensures that the interface has no IP address:

    1. ...
    2.     interfaces:
    3.     - name: eth1
    4.       description: No IP on eth1
    5.       type: ethernet
    6.       state: up
    7.       ipv4:
    8.         enabled: false
    9. ...

    Dynamic host configuration

    The following snippet configures an Ethernet interface that uses a dynamic IP address, gateway address, and DNS:

    1. ...
    2.     interfaces:
    3.     - name: eth1
    4.       description: DHCP on eth1
    5.       type: ethernet
    6.       state: up
    7.       ipv4:
    8.         dhcp: true
    9.         enabled: true
    10. ...

    The following snippet configures an Ethernet interface that uses a dynamic IP address but does not use a dynamic gateway address or DNS:

    1. ...
    2.     interfaces:
    3.     - name: eth1
    4.       description: DHCP without gateway or DNS on eth1
    5.       type: ethernet
    6.       state: up
    7.       ipv4:
    8.         dhcp: true
    9.         auto-gateway: false
    10.         auto-dns: false
    11.         enabled: true
    12. ...

    DNS

    The following snippet sets DNS configuration on the host.

    1. ...
    2.     interfaces:
    3.        ...
    4.     dns-resolver:
    5.       config:
    6.         search:
    7.         - example.com
    8.         - example.org
    9.         server:
    10.         - 8.8.8.8

    The following snippet configures a static route and a static IP on interface eth1.