Container image signatures

    Quay.io serves most of the images that make up OKD, and only the release image is signed. Release images refer to the approved OKD images, offering a degree of protection against supply chain attacks. However, some extensions to OKD, such as logging, monitoring, and service mesh, are shipped as Operators from the Operator Lifecycle Manager (OLM). Those images ship from the registry.

    To verify the integrity of those images between Red Hat registries and your infrastructure, enable signature verification.

    Enabling container signature validation for Red Hat Container Registries requires writing a signature verification policy file specifying the keys to verify images from these registries. For RHEL8 nodes, the registries are already defined in by default.

    Procedure

    1. Create a Butane config file, 51-worker-rh-registry-trust.bu, containing the necessary configuration for the worker nodes.

    2. Use Butane to generate a machine config YAML file, 51-worker-rh-registry-trust.yaml, containing the file to be written to disk on the worker nodes:

      1. $ butane 51-worker-rh-registry-trust.bu -o 51-worker-rh-registry-trust.yaml

    3. Apply the created machine config:

      1. $ oc apply -f 51-worker-rh-registry-trust.yaml

    4. Check that the worker machine config pool has rolled out with the new machine config:

      1. Check that the new machine config was created:

        1. $ oc get mc
        1. NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
        2. 00-master                                          a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        3. 00-worker                                          a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        4. 01-master-container-runtime                        a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        5. 01-master-kubelet                                  a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        6. 01-worker-container-runtime                        a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        7. 01-worker-kubelet                                  a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        8. 51-master-rh-registry-trust                                                                   3.2.0             13s
        9. 51-worker-rh-registry-trust                                                                   3.2.0             53s (1)
        10. 99-master-generated-crio-seccomp-use-default                                                  3.2.0             25m
        11. 99-master-generated-registries                     a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        12. 99-master-ssh                                                                                 3.2.0             28m
        13. 99-worker-generated-crio-seccomp-use-default                                                  3.2.0             25m
        14. 99-worker-generated-registries                     a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             25m
        15. 99-worker-ssh                                                                                 3.2.0             28m
        16. rendered-master-af1e7ff78da0a9c851bab4be2777773b   a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             8s
        17. rendered-master-cd51fd0c47e91812bfef2765c52ec7e6   a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             24m
        18. rendered-worker-2b52f75684fbc711bd1652dd86fd0b82   a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             24m
        19. rendered-worker-be3b3bce4f4aa52a62902304bac9da3c   a2178ad522c49ee330b0033bb5cb5ea132060b0a   3.2.0             48s (2)

      2. Check that the worker machine config pool is updating with the new machine config:

        1. $ oc get mcp

        Sample output

        1. NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
        2. master   rendered-master-af1e7ff78da0a9c851bab4be2777773b   True      False      False      3              3                   3                     0                      30m
        3. worker   rendered-worker-be3b3bce4f4aa52a62902304bac9da3c   False     True       False      3              0                   0                     0                      30m (1)

    5. If your cluster uses any RHEL7 worker nodes, when the worker machine config pool is updated, create YAML files on those nodes in the /etc/containers/registries.d directory, which specify the location of the detached signatures for a given registry server. The following example works only for images hosted in registry.access.redhat.com and registry.redhat.io.

      1. Start a debug session to each RHEL7 worker node:

      2. Change your root directory to /host:

        1. sh-4.2# chroot /host

      3. Create a /etc/containers/registries.d/registry.redhat.io.yaml file that contains the following:

        1. docker:
        2.      registry.redhat.io:
        3.          sigstore: https://registry.redhat.io/containers/sigstore

      4. Create a /etc/containers/registries.d/registry.access.redhat.com.yaml file that contains the following:

        1. docker:
        2.      registry.access.redhat.com:
        3.          sigstore: https://access.redhat.com/webassets/docker/content/sigstore

      5. Exit the debug session.

    After you apply the machine configs to the cluster, the Machine Config Controller detects the new MachineConfig object and generates a new rendered-worker-<hash> version.

    Prerequisites

    • You enabled signature verification by using a machine config file.

    1. On the command line, run the following command to display information about a desired worker:

      1. $ oc describe machineconfigpool/worker

      Example output of initial worker monitoring

      1. Name:         worker
      2. Namespace:
      3. Labels:       machineconfiguration.openshift.io/mco-built-in=
      4. Annotations:  <none>
      5. API Version:  machineconfiguration.openshift.io/v1
      6. Kind:         MachineConfigPool
      7. Metadata:
      8.   Creation Timestamp:  2019-12-19T02:02:12Z
      9.   Generation:          3
      10.   Resource Version:    16229
      11.   Self Link:           /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
      12.   UID:                 92697796-2203-11ea-b48c-fa163e3940e5
      13. Spec:
      14.   Configuration:
      15.     Source:
      16.       API Version:  machineconfiguration.openshift.io/v1
      18.       Name:         00-worker
      19.       API Version:  machineconfiguration.openshift.io/v1
      20.       Kind:         MachineConfig
      21.       Name:         01-worker-container-runtime
      22.       API Version:  machineconfiguration.openshift.io/v1
      23.       Kind:         MachineConfig
      24.       Name:         01-worker-kubelet
      25.       API Version:  machineconfiguration.openshift.io/v1
      26.       Kind:         MachineConfig
      27.       Name:         51-worker-rh-registry-trust
      28.       API Version:  machineconfiguration.openshift.io/v1
      29.       Kind:         MachineConfig
      30.       Name:         99-worker-92697796-2203-11ea-b48c-fa163e3940e5-registries
      31.       API Version:  machineconfiguration.openshift.io/v1
      32.       Kind:         MachineConfig
      33.       Name:         99-worker-ssh
      34.   Machine Config Selector:
      35.     Match Labels:
      36.       machineconfiguration.openshift.io/role:  worker
      37.   Node Selector:
      38.     Match Labels:
      39.       node-role.kubernetes.io/worker:
      40.   Paused:                              false
      41. Status:
      42.   Conditions:
      43.     Last Transition Time:  2019-12-19T02:03:27Z
      44.     Message:
      45.     Reason:
      46.     Status:                False
      47.     Type:                  RenderDegraded
      48.     Last Transition Time:  2019-12-19T02:03:43Z
      49.     Message:
      50.     Reason:
      51.     Status:                False
      52.     Type:                  NodeDegraded
      53.     Last Transition Time:  2019-12-19T02:03:43Z
      54.     Message:
      55.     Reason:
      56.     Status:                False
      57.     Type:                  Degraded
      58.     Last Transition Time:  2019-12-19T02:28:23Z
      59.     Message:
      60.     Reason:
      61.     Status:                False
      62.     Type:                  Updated
      63.     Last Transition Time:  2019-12-19T02:28:23Z
      64.     Message:               All nodes are updating to rendered-worker-f6819366eb455a401c42f8d96ab25c02
      65.     Reason:
      66.     Status:                True
      67.     Type:                  Updating
      68.   Configuration:
      69.     Name:  rendered-worker-d9b3f4ffcfd65c30dcf591a0e8cf9b2e
      70.     Source:
      71.       API Version:            machineconfiguration.openshift.io/v1
      72.       Kind:                   MachineConfig
      73.       Name:                   00-worker
      74.       API Version:            machineconfiguration.openshift.io/v1
      75.       Kind:                   MachineConfig
      76.       Name:                   01-worker-container-runtime
      77.       API Version:            machineconfiguration.openshift.io/v1
      78.       Kind:                   MachineConfig
      79.       Name:                   01-worker-kubelet
      80.       Kind:                   MachineConfig
      82.       API Version:            machineconfiguration.openshift.io/v1
      83.       Kind:                   MachineConfig
      84.       Name:                   99-worker-ssh
      85.   Degraded Machine Count:     0
      86.   Machine Count:              1
      87.   Observed Generation:        3
      88.   Ready Machine Count:        0
      89.   Unavailable Machine Count:  1
      90.   Updated Machine Count:      0
      91. Events:                       <none>

    2. Run the oc describe command again:

      1. $ oc describe machineconfigpool/worker

      Example output after the worker is updated

    3. Confirm that the policy.json file exists with the following command:

      1. $ oc debug node/<node> -- chroot /host cat /etc/containers/policy.json

      Example output

      1. Starting pod/<node>-debug ...
      2. To use host binaries, run `chroot /host`
      3. {
      4.   "default": [
      5.     {
      6.       "type": "insecureAcceptAnything"
      7.     }
      8.   ],
      9.   "transports": {
      10.     "docker": {
      11.       "registry.access.redhat.com": [
      12.         {
      13.           "type": "signedBy",
      14.           "keyType": "GPGKeys",
      15.           "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
      16.         }
      17.       ],
      18.       "registry.redhat.io": [
      19.         {
      20.           "type": "signedBy",
      21.           "keyType": "GPGKeys",
      22.           "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
      23.         }
      24.       ]
      25.     },
      26.     "docker-daemon": {
      27.       "": [
      28.         {
      29.           "type": "insecureAcceptAnything"
      30.         }
      31.       ]
      32.     }
      33.   }
      34. }

    4. Confirm that the registry.redhat.io.yaml file exists with the following command:

      1. $ oc debug node/<node> -- chroot /host cat /etc/containers/registries.d/registry.redhat.io.yaml

      Example output

      1. Starting pod/<node>-debug ...
      2. To use host binaries, run `chroot /host`
      3. docker:
      4.      registry.redhat.io:
      5.          sigstore: https://registry.redhat.io/containers/sigstore

    5. Confirm that the registry.access.redhat.com.yaml file exists with the following command:

      1. $ oc debug node/<node> -- chroot /host cat /etc/containers/registries.d/registry.access.redhat.com.yaml
      1. Starting pod/<node>-debug ...
      2. To use host binaries, run `chroot /host`
      3. docker:
      4.      registry.access.redhat.com: