Antrea API

    Starting with the v1.0 release, we decided to group all the Custom Resource Definitions (CRDs) defined by Antrea in a single API group, crd.antrea.io, instead of grouping CRDs logically in different API groups based on their purposes. The rationale for this change was to avoid proliferation of API groups. As a result, all resources in the crd.antrea.io are versioned individually, while before the v1.0 release, we used to have a single version number for all the CRDs in a given group: when introducing a new version of the API group, we would “move” all CRDs from the earlier version to the new version together. This explains why the tables below are presented differently for crd.antrea.io and for other API groups.

    For information about the Antrea API versioning policy, please refer to this document.

    These are the CRDs currently available in crd.antrea.io.

    These are the API group versions which are curently available when using Antrea.

    For the v1.0 release, we undertook to rename all Antrea APIs to use the antrea.io suffix instead of the antrea.tanzu.vmware.com suffix. For more information about the motivations behind this undertaking, please refer to Github issue #1715.

    From the v1.6 release, all legacy APIs (ending with the antrea.tanzu.vmware.com suffix) have been completely removed. If you are running an Antrea version older than v1.0 and you want to upgrade to Antrea v1.6 or greater and migrate your API resources, you will first need to do an intermediate upgrade to an Antrea version >= v1.0 and <= v1.5. You will then be able to migrate all your API resources to the new () API, by following the steps below. Finally, you will be able to upgrade to your desired Antrea version (>= v1.6).

    As part of the API renaming, and to avoid proliferation of API groups, we have decided to group all the Custom Resource Definitions (CRDs) defined by Antrea in a single API group: crd.antrea.io.

    1. As per our , older Agents need to be able to communicate with a new upgraded Controller, using the old controlplane.antrea.tanzu.vmware.com API. Once both the Controller and the Agent are upgraded, they communicate using controlplane.antrea.io.
    2. API Services can be accessed using either API version.
    3. For each resource in each API group, the new resource type should be backward-compatible with the old resource type, and, whenever possible, forward-compatible. This simplifies the upgrade of existing client applications which leverage the Antrea API. These applications can be easily upgraded to use the new API version, with no change to the business logic. Custom Resources created before upgrading the application can be accessed through the new API with no loss of information.

    To achieve our 3rd goal, we introduced a new Kubernetes controller in the Antrea Controller, in charge of mirroring “old” Custom Resources (created using the *.antrea.tanzu.vmware.com API groups) to the new (*.antrea.io) API. This new mirroring controller is enabled by default, but can be disabled by setting legacyCRDMirroring to false in the antrea-controller configuration options. Thanks to this controller, the Antrea components (Agent and Controller) only need to watch Custom Resources created with the new API group. If any client still uses the old (or “legacy”) API groups, these Custom Resources will be mirrored to the new API group and handled as expected.

    The mirroring controller behaves as follows:

    • If a Custom Resource is created with the legacy API, it will create a new Custom Resource with the same Spec and Labels as the legacy one.
    • Any update to the Spec and / or Labels of the legacy Custom Resource will be reflected identically in the new Custom Resource.
    • Any update to the Status of the new mirrored Custom Resource (assuming it has a Status field) will be reflected back identically in the legacy Custom Resource.
    • If the legacy Custom Resource is deleted, the mirrored one will be deleted automatically as well.
    • Manual updates to new mirrored Custom Resources will be overwritten by the controller.
    • If a legacy Custom Resource is annotated with "crd.antrea.io/stop-mirror", it will then be ignored, and updates to the corresponding new Custom Resource will no longer be overwritten.

    This gives us the following upgrade sequence for a client application which uses the legacy Antrea CRDs:

    1. Ensure that Antrea has been upgraded in the cluster to a version greater than or equal to v1.0, and that legacy CRD mirroring is enabled (this is the case by default).

    2. Stop the old version of the application, which uses the legacy CRDs.

    3. Annotate all existing Custom Resources managed by the application with "crd.antrea.io/stop-mirror". From now on, the mirroring controller will ignore these legacy resources: updates to the legacy resources (including deletions) are not applied to the corresponding new resource any more, and changes to the new resources are now possible (they will not be overwritten by the controller). As an example, the command below will annotate all ANPs in the current Namespace with "crd.antrea.io/stop-mirror".

      1. kubectl annotate lanp.security.antrea.tanzu.vmware.com --all crd.antrea.io/stop-mirror=''

    4. If you remove the filter, all your ANPs should still exist:

      1. kubectl get anp.crd.antrea.io -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

    5. Safely delete all legacy CRDs previously managed by the application. As an example, the command below will delete all legacy ANPs in the current Namespace:

      Once again, all new ANPs should still exist, which can be confirmed with:

      1. kubectl get anp.crd.antrea.io -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

    6. Start the new version of the application, which uses the new CRDs. All mirrored Custom Resources should be available for the application to access.

    Note that for CRDs which are “owned” by Antrea, AntreaAgentInfo and , resources are automatically created by the Antrea components using both API versions.

    After a successful upgrade from Antrea < v1.6 to Antrea >= v1.6, you may want to manually clean up legacy Kubernetes resources which were created by an old Antrea version but are no longer needed. Note that keeping these resource will not impact any Antrea functions.