Troubleshooting CNI plugin-related errors

    Service issues exist for pod CNI network setup and tear down in containerd v1.6.0-v1.6.3 when the CNI plugins have not been upgraded and/or the CNI config version is not declared in the CNI config files. The containerd team reports, “these issues are resolved in containerd v1.6.4.”

    With containerd v1.6.0-v1.6.3, if you do not upgrade the CNI plugins and/or declare the CNI config version, you might encounter the following “Incompatible CNI versions” or “Failed to destroy network for sandbox” error conditions.

    If the version of your CNI plugin does not correctly match the plugin version in the config because the config version is later than the plugin version, the containerd log will likely show an error message on startup of a pod similar to:

    If the version of the plugin is missing in the CNI plugin config, the pod may run. However, stopping the pod generates an error similar to:

    This error leaves the pod in the not-ready state with a network namespace still attached. To recover from this problem, to add the missing version information. The next attempt to stop the pod should be successful.

    If you’re using containerd v1.6.0-v1.6.3 and encountered “Incompatible CNI versions” or “Failed to destroy network for sandbox” errors, consider updating your CNI plugins and editing the CNI config files.

    1. .
    2. After stopping your container runtime and kubelet services, perform the following upgrade operations:
    • If you’re running CNI plugins, upgrade them to the latest version.
    • If you’re using non-CNI plugins, replace them with CNI plugins. Use the latest version of the plugins.
    • For , ensure that you have installed the latest version (v1.0.0 or later) of the CNI loopback plugin.
    • Upgrade node components (for example, the kubelet) to Kubernetes v1.24
    • Upgrade to or install the most current version of the container runtime.
    1. Bring the node back into your cluster by restarting your container runtime and kubelet. Uncordon the node (kubectl uncordon <nodename>).

    An example containerd configuration file

    The following example shows a configuration for containerd runtime v1.6.x, which supports a recent version of the CNI specification (v1.0.0).

    Please see the documentation from your plugin and networking provider for further instructions on configuring your system.

    On Kubernetes, containerd runtime adds a loopback interface, lo, to pods as a default behavior. The containerd runtime configures the loopback interface via a CNI plugin, . The loopback plugin is distributed as part of the containerd release packages that have the cni designation. v1.6.0 and later includes a CNI v1.0.0-compatible loopback plugin as well as other default CNI plugins. The configuration for the loopback plugin is done internally by containerd, and is set to use CNI v1.0.0. This also means that the version of the loopback plugin must be v1.0.0 or later when this newer version containerd is started.

    Update the IP address ranges in the preceding example with ones that are based on your use case and network addressing plan.