Transparent Encryption (stable/beta)

    Note

    The encryption feature is stable in combination with the direct-routing and ENI datapath mode. In combination with encapsulation/tunneling, the feature is still in beta phase.

    Packets destined to the same node they were sent out of are not encrypted. This is a intended behavior as it doesn’t provide any benefits because the raw traffic on the node can be seen.

    Transparent encryption is not currently supported when chaining Cilium on top of other CNI plugins. For more information, see GitHub issue #15596.

    Note

    resources need to be deployed in the same namespace as Cilium! In our example, we use kube-system.

    First, create a Kubernetes secret for the IPsec keys to be stored. This will generate the necessary IPsec keys which will be distributed as a Kubernetes secret called cilium-ipsec-keys. In this example we use GMC-128-AES, but any of the supported Linux algorithms may be used. To generate, use the following:

    The secret can be seen with kubectl -n kube-system get secret and will be listed as “cilium-ipsec-keys”.

    1. $ kubectl -n kube-system get secrets cilium-ipsec-keys
    2. NAME                TYPE     DATA   AGE
    3. cilium-ipsec-keys   Opaque   1      176m

    Enable Encryption in Cilium

    Note

    First, make sure you have Helm 3 installed.

    Setup Helm repository:

    Deploy Cilium release via Helm with the following options to enable encryption:

    1. helm install cilium cilium/cilium --version 1.8.10 \
    2.   --namespace kube-system \
    3.   --set global.encryption.enabled=true \
    4.   --set global.encryption.nodeEncryption=false

    These options can be provided along with other options, such as when deploying to GKE, with VXLAN tunneling:

    1. helm install cilium cilium/cilium --version 1.8.10 \
    2.   --namespace cilium   --set global.nodeinit.enabled=true   --set nodeinit.reconfigureKubelet=true   --set nodeinit.removeCbrBridge=true   --set global.cni.binPath=/home/kubernetes/bin   --set global.tunnel=vxlan   --set global.encryption.enabled=true   --set global.encryption.nodeEncryption=false

    At this point the Cilium managed nodes will be using IPsec for all traffic. For further information on Cilium’s transparent encryption, see .

    If direct routing is being used, an additional argument can be used to identify the network-facing interface. If no interface is specified, the default route link is chosen by inspecting the routing tables. This will work in many cases, but depending on routing rules, users may need to specify the encryption interface as follows:

    Node to node encryption

    In order to enable node-to-node encryption, add:

    1. [...]
    2. --set global.encryption.enabled=true \
    3. --set global.encryption.nodeEncryption=true
    4. --set global.tunnel=disabled

    Note

    Node to node encryption feature is tested and supported with direct routing modes. Using with encapsulation/tunneling is not currently tested or supported.

    Support with tunneling mode is tracked at .

    Run a bash shell in one of the Cilium pods with kubectl -n <k8s namespace> exec -ti <cilium pod> -- bash and execute the following commands:

    1. Install tcpdump
    1. apt-get update
    2. apt-get -y install tcpdump
    1. Check that traffic is encrypted:
    1. tcpdump -n -i cilium_vxlan
    2. listening on cilium_vxlan, link-type EN10MB (Ethernet), capture size 262144 bytes
    3. 15:16:21.626416 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e2), length 180
    5. 15:16:21.627167 IP 10.60.0.1 > 10.60.1.1: ESP(spi=0x00000001,seq=0x579d), length 100
    6. 15:16:21.627296 IP 10.60.0.1 > 10.60.1.1: ESP(spi=0x00000001,seq=0x579e), length 100
    7. 15:16:21.627523 IP 10.60.0.1 > 10.60.1.1: ESP(spi=0x00000001,seq=0x579f), length 180
    8. 15:16:21.627699 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e4), length 100
    9. 15:16:21.628408 IP 10.60.1.1 > 10.60.0.1: ESP(spi=0x00000001,seq=0x57e5), length 100

    Key Rotation

    1. KEYID=$(kubectl get secret -n kube-system cilium-ipsec-keys -o yaml | awk '/^\s*keys:/ {print $2}' | base64 -d | awk '{print $1}')
    2. if [[ $KEYID -gt 15 ]]; then KEYID=0; fi
    3. data=$(echo "{\"stringData\":{\"keys\":\"$((($KEYID+1))) "rfc4106\(gcm\(aes\)\)" $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null| xxd -p -c 64)) 128\"}}")
    4. kubectl patch secret -n kube-system cilium-ipsec-keys -p="${data}" -v=1

    Then restart Cilium agents to transition to the new key. During transition the new and old keys will be in use. The Cilium agent keeps per endpoint data on which key is used by each endpoint and will use the correct key if either side has not yet been updated. In this way encryption will work as new keys are rolled out.

    The KEYID environment variable in the above example stores the current key ID used by Cilium. The key variable is a uint8 with value between 0-16 and should be monotonically increasing every re-key with a rollover from 16 to 0. The Cilium agent will default to KEYID of zero if its not specified in the secret.

    • Make sure that the Cilium pods have kvstore connectivity:

    • Check for level=warning and level=error messages in the Cilium log files

    • Run a bash in a Cilium and validate the following:

      • Routing rules matching on fwmark:

        1. ip rule list
        2. 1:      from all fwmark 0xd00/0xf00 lookup 200
        3. 1:      from all fwmark 0xe00/0xf00 lookup 200
        4. [...]

      • Content of routing table 200

        1. ip route list table 200
        2. local 10.60.0.0/24 dev cilium_vxlan proto 50 scope host
        3. 10.60.1.0/24 via 10.60.0.1 dev cilium_host

      • XFRM policy:

        1. ip xfrm p
        2. src 10.60.1.1/24 dst 10.60.0.1/24
        3.         mark 0xd00/0xf00
        4.         tmpl src 10.60.1.1 dst 10.60.0.1
        5.                 proto esp spi 0x00000001 reqid 1 mode tunnel
        7.         dir in priority 0
        8.         mark 0xd00/0xf00
        9.         tmpl src 10.60.1.1 dst 10.60.0.1
        10.                 proto esp spi 0x00000001 reqid 1 mode tunnel
        11. src 10.60.0.1/24 dst 10.60.1.1/24
        12.         dir out priority 0
        13.         mark 0xe00/0xf00
        14.         tmpl src 10.60.0.1 dst 10.60.1.1
        15.                 proto esp spi 0x00000001 reqid 1 mode tunnel

      • XFRM state:

        1. ip xfrm s
        2. src 10.60.0.1 dst 10.60.1.1
        3.         proto esp spi 0x00000001 reqid 1 mode tunnel
        4.         replay-window 0
        5.         auth-trunc hmac(sha256) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546 96
        6.         enc cbc(aes) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546
        7.         anti-replay context: seq 0x0, oseq 0xe0c0, bitmap 0x00000000
        8.         sel src 0.0.0.0/0 dst 0.0.0.0/0
        9. src 10.60.1.1 dst 10.60.0.1
        10.         proto esp spi 0x00000001 reqid 1 mode tunnel
        11.         replay-window 0
        12.         auth-trunc hmac(sha256) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546 96
        13.         enc cbc(aes) 0x6162636465666768696a6b6c6d6e6f70717273747576777a797a414243444546
        14.         sel src 0.0.0.0/0 dst 0.0.0.0/0

    Disabling Encryption

    To disable the encryption, regenerate the YAML with the option global.encryption.enabled=false