Egress

    What is Egress?

    You may be interested in using this capability if any of the following apply:

    • A consistent IP address is desired when specific Pods connect to services outside of the cluster, for source tracing in audit logs, or for filtering by source IP in external firewall, etc.

    • You want to force outgoing external connections to leave the cluster via certain Nodes, for security controls, or due to network topology restrictions.

    This guide demonstrates how to configure Egress to achieve the above result.

    Egress was introduced in v1.0 as an alpha feature, and was graduated to beta in v1.6, at which time it was enabled by default. Prior to v1.6, a feature gate, Egress must be enabled on the antrea-controller and antrea-agent in the antrea-config ConfigMap like the following options for the feature to work:

    The Egress resource

    A typical Egress resource example:

    1. apiVersion: crd.antrea.io/v1alpha2
    2. kind: Egress
    3. metadata:
    4.   name: egress-prod-web
    5. spec:
    6.   appliedTo:
    7.     namespaceSelector:
    8.       matchLabels:
    9.         env: prod
    10.     podSelector:
    11.       matchLabels:
    12.         role: web
    13.   egressIP: 10.10.0.8 # can be populated by Antrea after assigning an IP from the pool below
    14.   externalIPPool: prod-external-ip-pool
    15.   egressNode: node01

    The appliedTo field specifies the grouping criteria of Pods to which the Egress applies to. Pods can be selected cluster-wide using podSelector. If set with a namespaceSelector, all Pods from Namespaces selected by the namespaceSelector will be selected. Specific Pods from specific Namespaces can be selected by providing both a podSelector and a namespaceSelector. Empty appliedTo selects nothing. The field is mandatory.

    EgressIP

    The egressIP field specifies the egress (SNAT) IP the traffic from the selected Pods to the external network should use. The IP must be reachable from all Nodes. The IP can be specified when creating the Egress. Starting with Antrea v1.2, it can be allocated from an ExternalIPPool automatically.

    • If egressIP is not specified, externalIPPool must be specified. An IP will be allocated from the pool by the antrea-controller. The IP will be assigned to a Node selected by the nodeSelector of the externalIPPool automatically.
    • If both egressIP and externalIPPool are specified, the IP must be in the range of the pool. Similarly, the IP will be assigned to a Node selected by the externalIPPool automatically.
    • If only is specified, Antrea will not manage the assignment of the IP and it must be assigned to an arbitrary interface of one Node manually.

    Note: If more than one Egress applies to a Pod and they specify different egressIP, the effective egress IP will be selected randomly.

    The externalIPPool field specifies the name of the ExternalIPPool that the egressIP should be allocated from. It also determines which Nodes the IP can be assigned to. It can be empty, which means users should assign the egressIP to one Node manually.

    ExternalIPPool defines one or multiple IP ranges that can be used in the external network. The IPs in the pool can be allocated to the Egress resources as the Egress IPs. A typical ExternalIPPool resource example:

    IPRanges

    The ipRanges field contains a list of IP ranges representing the available IPs of this IP pool. Each IP range may consist of a cidr or a pair of start and end IPs (which are themselves included in the range).

    The nodeSelector field specifies which Nodes the IPs in this pool can be assigned to. It’s useful when you want to limit egress traffic to certain Nodes. The semantics of the selector is the same as those used elsewhere in Kubernetes, i.e. both matchLabels and matchExpressions are supported. It can be empty, which means all Nodes can be selected.

    Usage examples

    Configuring High-Availability Egress

    In this example, we will make web apps in different namespaces use different egress IPs to access the external network.

    First, create an ExternalIPPool with a list of external routable IPs on the network.

    1. apiVersion: crd.antrea.io/v1alpha2
    2. kind: ExternalIPPool
    3. metadata:
    4.   name: external-ip-pool
    5. spec:
    6.   ipRanges:
    7.   - start: 10.10.0.11  # 10.10.0.11-10.10.0.20 can be used as Egress IPs
    8.     end: 10.10.0.20
    9.   nodeSelector: {}     # All Nodes can be Egress Nodes

    Then create two Egress resources, each of which applies to web apps in one Namespace.

    1. # kubectl get egress
    2. NAME                 EGRESSIP       AGE   NODE
    3. egress-prod-web      10.10.0.11     1m    node-4
    4. egress-staging-web   10.10.0.12     1m    node-6

    Now, the packets from the Pods with label app=web in the Namespace to the external network will be redirected to the node-4 Node and SNATed to 10.10.0.11 while the packets from the Pods with label app=web in the staging Namespace to the external network will be redirected to the node-6 Node and SNATed to 10.10.0.12.

    Finally, if the node-4 Node powers off, 10.10.0.11 will be re-assigned to another available Node quickly, and the packets from the Pods with label app=web in the prod Namespace will be redirected to the new Node, minimizing egress connection disruption without manual intervention.

    In this example, we will make Pods in different namespaces use specific Node IPs (or any IPs that are configured to the interfaces of the Nodes) to access the external network.

    Since the Egress IPs have been configured to the Nodes, we can create Egress resources with specific IPs directly.

    List the Egress resource with kubectl. The output shows 10.10.0.104 is discovered on node-4 Node while 10.10.0.105 is discovered on node-5.

    1. # kubectl get egress
    2. NAME                 EGRESSIP       AGE   NODE
    3. egress-prod          10.10.0.104    1m    node-4
    4. egress-staging       10.10.0.105    1m    node-5

    Now, the packets from the Pods with in the prod Namespace to the external network will be redirected to the node-4 Node and SNATed to 10.10.0.104 while the packets from the Pods in the staging Namespace to the external network will be redirected to the node-5 Node and SNATed to 10.10.0.105.

    In this configuration, if the node-4 Node powers off, re-configuring 10.10.0.104 to another Node or updating the egressIP of egress-prod to another Node’s IP can recover the egress connection. Antrea will detect the configuration change and redirect the packets from the Pods in the prod Namespace to the new Node.

    This feature is currently only supported for Nodes running Linux and “encap” mode. The support for Windows and other traffic modes will be added in the future.