Manually installing a single-node OpenShift cluster with ZTP

    The target bare-metal host must meet the networking, firmware, and hardware requirements listed in Recommended cluster configuration for vDU application workloads.

    Use the generator entrypoint for the ztp-site-generate container to generate the site installation and configuration custom resource (CRs) for a cluster based on SiteConfig and PolicyGenTemplate CRs.

    Prerequisites

    • You have installed the OpenShift CLI (oc).

    • You have logged in to the hub cluster as a user with cluster-admin privileges.

    Procedure

    1. Create an output folder by running the following command:

    2. Export the argocd directory from the ztp-site-generate container image:

      1. $ podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.13 extract /home/ztp --tar | tar x -C ./out

      The ./out directory has the reference PolicyGenTemplate and SiteConfig CRs in the out/argocd/example/ folder.

      Example output

      1. out
      2.  └── argocd
      3.       └── example
      4.            ├── policygentemplates
      5.            │     ├── common-ranGen.yaml
      6.            │     ├── example-sno-site.yaml
      7.            │     ├── group-du-sno-ranGen.yaml
      8.            │     ├── group-du-sno-validator-ranGen.yaml
      9.            │     ├── kustomization.yaml
      10.            │     └── ns.yaml
      11.            └── siteconfig
      12.                   ├── example-sno.yaml
      13.                   ├── KlusterletAddonConfigOverride.yaml
      14.                   └── kustomization.yaml

    3. Create an output folder for the site installation CRs:

      1. $ mkdir -p ./site-install

    4. Modify the example SiteConfig CR for the cluster type that you want to install. Copy example-sno.yaml to site-1-sno.yaml and modify the CR to match the details of the site and bare-metal host that you want to install, for example:

      Example single-node OpenShift cluster SiteConfig CR

      1. apiVersion: ran.openshift.io/v1
      2. kind: SiteConfig
      3. metadata:
      4.   name: "<site_name>"
      5.   namespace: "<site_name>"
      6. spec:
      7.   baseDomain: "example.com"
      8.   pullSecretRef:
      9.     name: "assisted-deployment-pull-secret" (1)
      10.   clusterImageSetNameRef: "openshift-4.13" (2)
      11.   sshPublicKey: "ssh-rsa AAAA..." (3)
      12.   clusters:
      13.   - clusterName: "<site_name>"
      14.     networkType: "OVNKubernetes"
      15.     clusterLabels: (4)
      16.       common: true
      17.       group-du-sno: ""
      18.       sites : "<site_name>"
      19.     clusterNetwork:
      20.       - cidr: 1001:1::/48
      21.         hostPrefix: 64
      22.     machineNetwork:
      23.       - cidr: 1111:2222:3333:4444::/64
      24.     serviceNetwork:
      25.       - 1001:2::/112
      26.     additionalNTPSources:
      27.       - 1111:2222:3333:4444::2
      28.     #crTemplates:
      29.     #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml" (5)
      30.     nodes:
      31.       - hostName: "example-node.example.com" (6)
      32.         role: "master"
      33.         bmcAddress: idrac-virtualmedia://<out_of_band_ip>/<system_id>/ (7)
      34.         bmcCredentialsName:
      35.           name: "bmh-secret" (8)
      36.         bootMACAddress: "AA:BB:CC:DD:EE:11"
      37.         bootMode: "UEFI" (9)
      38.         rootDeviceHints:
      39.           wwn: "0x11111000000asd123"
      40.         cpuset: "0-1,52-53"  (10)
      41.         nodeNetwork: (11)
      42.           interfaces:
      43.             - name: eno1
      44.               macAddress: "AA:BB:CC:DD:EE:11"
      45.           config:
      46.             interfaces:
      47.               - name: eno1
      48.                 type: ethernet
      49.                 state: up
      50.                 ipv4:
      51.                   enabled: false
      52.                 ipv6: (12)
      53.                   enabled: true
      54.                   address:
      55.                   - ip: 1111:2222:3333:4444::aaaa:1
      56.                     prefix-length: 64
      57.             dns-resolver:
      58.               config:
      59.                 search:
      60.                 - example.com
      61.                 server:
      62.                 - 1111:2222:3333:4444::2
      63.             routes:
      64.               config:
      65.               - destination: ::/0
      66.                 next-hop-interface: eno1
      67.                 next-hop-address: 1111:2222:3333:4444::1
      68.                 table-id: 254
      1Create the assisted-deployment-pull-secret CR with the same namespace as the SiteConfig CR.
      2clusterImageSetNameRef defines an image set available on the hub cluster. To see the list of supported versions on your hub cluster, run oc get clusterimagesets.
      3Configure the SSH public key used to access the cluster.
      4Cluster labels must correspond to the bindingRules field in the PolicyGenTemplate CRs that you define. For example, policygentemplates/common-ranGen.yaml applies to all clusters with common: true set, policygentemplates/group-du-sno-ranGen.yaml applies to all clusters with group-du-sno: “” set.
      5Optional. The CR specifed under KlusterletAddonConfig is used to override the default KlusterletAddonConfig that is created for the cluster.
      6For single-node deployments, define a single host. For three-node deployments, define three hosts. For standard deployments, define three hosts with role: master and two or more hosts defined with role: worker.
      7BMC address that you use to access the host. Applies to all cluster types.
      8Name of the bmh-secret CR that you separately create with the host BMC credentials. When creating the bmh-secret CR, use the same namespace as the SiteConfig CR that provisions the host.
      9Configures the boot mode for the host. The default value is UEFI. Use UEFISecureBoot to enable secure boot on the host.
      10cpuset must match the value set in the cluster PerformanceProfile CR spec.cpu.reserved field for workload partitioning.
      11Specifies the network settings for the node.
      12Configures the IPv6 address for the host. For single-node OpenShift clusters with static IP addresses, the node-specific API and Ingress IPs should be the same.

    5. Generate the day-0 installation CRs by processing the modified CR site-1-sno.yaml by running the following command:

        Example output

        1. site-install
        2. └── site-1-sno
        3.     ├── site-1_agentclusterinstall_example-sno.yaml
        4.     ├── site-1-sno_baremetalhost_example-node1.example.com.yaml
        5.     ├── site-1-sno_clusterdeployment_example-sno.yaml
        6.     ├── site-1-sno_configmap_example-sno.yaml
        7.     ├── site-1-sno_infraenv_example-sno.yaml
        8.     ├── site-1-sno_klusterletaddonconfig_example-sno.yaml
        9.     ├── site-1-sno_machineconfig_02-master-workload-partitioning.yaml
        10.     ├── site-1-sno_machineconfig_predefined-extra-manifests-master.yaml
        11.     ├── site-1-sno_machineconfig_predefined-extra-manifests-worker.yaml
        12.     ├── site-1-sno_managedcluster_example-sno.yaml
        13.     ├── site-1-sno_namespace_example-sno.yaml
        14.     └── site-1-sno_nmstateconfig_example-node1.example.com.yaml

      1. Optional: Generate just the day-0 MachineConfig installation CRs for a particular cluster type by processing the reference SiteConfig CR with the -E option. For example, run the following commands:

        1. Create an output folder for the MachineConfig CRs:

          1. $ mkdir -p ./site-machineconfig

        2. Generate the MachineConfig installation CRs:

          1. $ podman run -it --rm -v `pwd`/out/argocd/example/siteconfig:/resources:Z -v `pwd`/site-machineconfig:/output:Z,U registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.13.1 generator install -E site-1-sno.yaml /output

          Example output

          1. site-machineconfig
          2. └── site-1-sno
          3.     ├── site-1-sno_machineconfig_02-master-workload-partitioning.yaml
          4.     ├── site-1-sno_machineconfig_predefined-extra-manifests-master.yaml
          5.     └── site-1-sno_machineconfig_predefined-extra-manifests-worker.yaml

      2. Generate and export the day-2 configuration CRs using the reference PolicyGenTemplate CRs from the previous step. Run the following commands:

        1. Create an output folder for the day-2 CRs:

          1. $ mkdir -p ./ref

        2. Generate and export the day-2 configuration CRs:

          1. $ podman run -it --rm -v `pwd`/out/argocd/example/policygentemplates:/resources:Z -v `pwd`/ref:/output:Z,U registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.13.1 generator config -N . /output

          The command generates example group and site-specific PolicyGenTemplate CRs for single-node OpenShift, three-node clusters, and standard clusters in the ./ref folder.

          Example output

          1. ref
          2.  └── customResource
          3.       ├── common
          4.       ├── example-multinode-site
          5.       ├── example-sno
          6.       ├── group-du-3node
          7.       ├── group-du-3node-validator
          8.       │    └── Multiple-validatorCRs
          9.       ├── group-du-sno
          10.       ├── group-du-sno-validator
          11.       ├── group-du-standard
          12.       └── group-du-standard-validator
          13.            └── Multiple-validatorCRs

      3. Use the generated CRs as the basis for the CRs that you use to install the cluster. You apply the installation CRs to the hub cluster as described in “Installing a single managed cluster”. The configuration CRs can be applied to the cluster after cluster installation is complete.

      Additional resources

      Creating the managed bare-metal host secrets

      Add the required Secret custom resources (CRs) for the managed bare-metal host to the hub cluster. You need a secret for the GitOps Zero Touch Provisioning (ZTP) pipeline to access the Baseboard Management Controller (BMC) and a secret for the assisted installer service to pull cluster installation images from the registry.

      The secrets are referenced from the SiteConfig CR by name. The namespace must match the SiteConfig namespace.

      Procedure

      1. Create a YAML secret file containing credentials for the host Baseboard Management Controller (BMC) and a pull secret required for installing OpenShift and all add-on cluster Operators:

        1. Save the following YAML as the file example-sno-secret.yaml:

          1. apiVersion: v1
          2. kind: Secret
          3. metadata:
          4.   name: example-sno-bmc-secret
          5.   namespace: example-sno (1)
          6. data: (2)
          7.   password: <base64_password>
          8.   username: <base64_username>
          9. type: Opaque
          10. ---
          11. apiVersion: v1
          12. kind: Secret
          13. metadata:
          14.   name: pull-secret
          15.   namespace: example-sno  (3)
          16. data:
          17.   .dockerconfigjson: <pull_secret> (4)
          18. type: kubernetes.io/dockerconfigjson

      2. Add the relative path to example-sno-secret.yaml to the kustomization.yaml file that you use to install the cluster.

      Configuring Discovery ISO kernel arguments for manual installations using GitOps ZTP

      The GitOps Zero Touch Provisioning (ZTP) workflow uses the Discovery ISO as part of the OKD installation process on managed bare-metal hosts. You can edit the InfraEnv resource to specify kernel arguments for the Discovery ISO. This is useful for cluster installations with specific environmental requirements. For example, configure the rd.net.timeout.carrier kernel argument for the Discovery ISO to facilitate static networking for the cluster or to receive a DHCP address before downloading the root file system during installation.

      In OKD 4.13, you can only add kernel arguments. You can not replace or delete kernel arguments.

      Prerequisites

      • You have installed the OpenShift CLI (oc).

      • You have logged in to the hub cluster as a user with cluster-admin privileges.

      • You have manually generated the installation and configuration custom resources (CRs).

      Procedure

      1. Edit the spec.kernelArguments specification in the InfraEnv CR to configure kernel arguments:
      1. apiVersion: agent-install.openshift.io/v1beta1
      2. kind: InfraEnv
      3. metadata:
      4.   name: <cluster_name>
      5.   namespace: <cluster_name>
      6. spec:
      7.   kernelArguments:
      8.     - operation: append (1)
      9.       value: audit=0 (2)
      10.     - operation: append
      11.       value: trace=1
      12.   clusterRef:
      13.     name: <cluster_name>
      14.     namespace: <cluster_name>
      15.   pullSecretRef:
      16.     name: pull-secret
      1Specify the append operation to add a kernel argument.
      2Specify the kernel argument you want to configure. This example configures the audit kernel argument and the trace kernel argument.

      The SiteConfig CR generates the InfraEnv resource as part of the day-0 installation CRs.

      Verification

      To verify that the kernel arguments are applied, after the Discovery image verifies that OKD is ready for installation, you can SSH to the target host before the installation process begins. At that point, you can view the kernel arguments for the Discovery ISO in the /proc/cmdline file.

      1. Begin an SSH session with the target host:

      2. View the system’s kernel arguments by using the following command:

        1. $ cat /proc/cmdline

      You can manually deploy a single managed cluster using the assisted service and Red Hat Advanced Cluster Management (RHACM).

      Prerequisites

      • You have installed the OpenShift CLI (oc).

      • You have logged in to the hub cluster as a user with cluster-admin privileges.

      • You have created the baseboard management controller (BMC) Secret and the image pull-secret Secret custom resources (CRs). See “Creating the managed bare-metal host secrets” for details.

      Procedure

      1. Create a ClusterImageSet for each specific cluster version to be deployed, for example clusterImageSet-4.13.yaml. A ClusterImageSet has the following format:

        1. apiVersion: hive.openshift.io/v1
        2. kind: ClusterImageSet
        3. metadata:
        4.   name: openshift-4.13.0 (1)
        5. spec:
        6.    releaseImage: quay.io/openshift-release-dev/ocp-release:4.13.0-x86_64 (2)

      2. Create the Namespace CR in the cluster-namespace.yaml file:

        1. apiVersion: v1
        2. kind: Namespace
        3. metadata:
        4.      name: <cluster_name> (1)
        6.         name: <cluster_name> (1)
        1The name of the managed cluster to provision.

      3. Apply the Namespace CR by running the following command:

        1. $ oc apply -f cluster-namespace.yaml

      4. Apply the generated day-0 CRs that you extracted from the ztp-site-generate container and customized to meet your requirements:

        1. $ oc apply -R ./site-install/site-sno-1

      Additional resources

      Monitoring the managed cluster installation status

      Ensure that cluster provisioning was successful by checking the cluster status.

      Prerequisites

      • All of the custom resources have been configured and provisioned, and the Agent custom resource is created on the hub for the managed cluster.

      Procedure

      1. Check the status of the managed cluster:

        1. $ oc get managedcluster

        True indicates the managed cluster is ready.

      2. Check the agent status:

        1. $ oc get agent -n <cluster_name>

      3. Use the describe command to provide an in-depth description of the agent’s condition. Statuses to be aware of include BackendError, InputError, ValidationsFailing, InstallationFailed, and AgentIsConnected. These statuses are relevant to the Agent and AgentClusterInstall custom resources.

        1. $ oc describe agent -n <cluster_name>

      4. Check the cluster provisioning status:

        1. $ oc get agentclusterinstall -n <cluster_name>

      5. Use the describe command to provide an in-depth description of the cluster provisioning status:

        1. $ oc describe agentclusterinstall -n <cluster_name>

      6. Check the status of the managed cluster’s add-on services:

        1. $ oc get managedclusteraddon -n <cluster_name>

      7. Retrieve the authentication information of the kubeconfig file for the managed cluster:

        1. $ oc get secret -n <cluster_name> <cluster_name>-admin-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d > <directory>/<cluster_name>-kubeconfig

      Troubleshooting the managed cluster

      Use this procedure to diagnose any installation issues that might occur with the managed cluster.

      Procedure

      1. Check the status of the managed cluster:

        1. $ oc get managedcluster

        Example output

        If the status in the AVAILABLE column is True, the managed cluster is being managed by the hub.

        If the status in the AVAILABLE column is Unknown, the managed cluster is not being managed by the hub. Use the following steps to continue checking to get more information.

      2. Check the AgentClusterInstall install status:

        1. $ oc get clusterdeployment -n <cluster_name>

        Example output

        1. NAME        PLATFORM            REGION   CLUSTERTYPE   INSTALLED    INFRAID    VERSION  POWERSTATE AGE
        2. Sno0026    agent-baremetal                               false                          Initialized
        3. 2d14h

        If the status in the INSTALLED column is false, the installation was unsuccessful.

      3. If the installation failed, enter the following command to review the status of the AgentClusterInstall resource:

        1. $ oc describe agentclusterinstall -n <cluster_name> <cluster_name>

      4. Resolve the errors and reset the cluster:

        1. Remove the cluster’s managed cluster resource:

          1. $ oc delete managedcluster <cluster_name>

        2. Remove the cluster’s namespace:

          1. $ oc delete namespace <cluster_name>

          This deletes all of the namespace-scoped custom resources created for this cluster. You must wait for the ManagedCluster CR deletion to complete before proceeding.

        3. Recreate the custom resources for the managed cluster.

      You can use the OKD web console to install Logical volume manager storage (LVM Storage).

      Prerequisites

      • Install the latest version of the RHACM Operator.

      • Log in as a user with cluster-admin privileges.

      Procedure

      1. In the OKD web console, navigate to Operators → OperatorHub.

      2. Search for the LVM Storage from the list of available Operators, and then click Install.

      3. Keep the default selection of Installation mode (“All namespaces on the cluster (default)”) and Installed Namespace (“openshift-operators”) to ensure that the Operator is installed properly.

      4. Click Install.

      Verification

      1. To confirm that the installation is successful:

        1. Navigate to the Operators → Installed Operators page.

        2. Check that the Operator is installed in the All Namespaces namespace and its status is Succeeded.

      2. If the Operator is not installed successfully:

        1. Navigate to the Operators → Installed Operators page and inspect the Status column for any errors or failures.

        2. Navigate to the Workloads → Pods page and check the logs in any containers in the local-storage-operator pod that are reporting issues.

      Additional resources

      Installing LVM Storage by using the CLI

      You can use the OpenShift CLI (oc) to install LVM Storage.

      Prerequisites

      • Install the latest version of the RHACM Operator.

      • Log in as a user with cluster-admin privileges.

      Procedure

      1. Create the openshift-storage namespace by running the following command:

        1. $ oc create ns openshift-storage

      2. Create an OperatorGroup CR.

        1. Define the OperatorGroup CR and save the YAML file, for example, lmvs-operatorgroup.yaml:

          Example OperatorGroup CR

          1. apiVersion: operators.coreos.com/v1
          2. kind: OperatorGroup
          3. metadata:
          4.   name: lvms-operator-operatorgroup
          5.   namespace: openshift-storage
          6.   annotations:
          7.     ran.openshift.io/ztp-deploy-wave: "2"
          8. spec:
          9.   targetNamespaces:
          10.   - openshift-storage

        2. Create the OperatorGroup CR by running the following command:

          1. $ oc create -f lmvs-operatorgroup.yaml

      3. Create a Subscription CR.

        1. Define the Subscription CR and save the YAML file, for example, lvms-subscription.yaml:

          Example Subscription CR

          1. apiVersion: operators.coreos.com/v1alpha1
          2. kind: Subscription
          3. metadata:
          4.   name: lvms-operator
          5.   namespace: openshift-storage
          6.   annotations:
          7.     ran.openshift.io/ztp-deploy-wave: "2"
          8. spec:
          9.   channel: "stable-4.13"
          10.   name: lvms-operator
          11.   source: redhat-operators
          12.   sourceNamespace: openshift-marketplace
          13.   installPlanApproval: Manual

        2. Create the Subscription CR by running the following command:

          1. $ oc create -f lvms-subscription.yaml

      Verification

      1. Verify that the installation succeeded by inspecting the CSV resource:

        1. $ oc get csv -n openshift-storage

        Example output

        1. NAME                                                   DISPLAY                            VERSION               REPLACES                           PHASE
        2. lvms-operator.4.13.x                                   LVM Storage                        4.13x                                                   Succeeded

      2. Verify that LVM Storage is up and running:

        1. $ oc get deploy -n openshift-storage

        Example output

        1. NAMESPACE                                          NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
        2. openshift-storage                                  lvms-operator                                    1/1     1            1           14s

      RHACM generated cluster installation CRs reference

      Red Hat Advanced Cluster Management (RHACM) supports deploying OKD on single-node clusters, three-node clusters, and standard clusters with a specific set of installation custom resources (CRs) that you generate using SiteConfig CRs for each site.

      Every managed cluster has its own namespace, and all of the installation CRs except for ManagedCluster and ClusterImageSet are under that namespace. ManagedCluster and ClusterImageSet are cluster-scoped, not namespace-scoped. The namespace and the CR names match the cluster name.

      The following table lists the installation CRs that are automatically applied by the RHACM assisted service when it installs clusters using the SiteConfig CRs that you configure.

      Table 1. Cluster installation CRs generated by RHACM
      CRDescriptionUsage

      BareMetalHost

      Contains the connection information for the Baseboard Management Controller (BMC) of the target bare-metal host.

      Provides access to the BMC to load and start the discovery image on the target server by using the Redfish protocol.

      InfraEnv

      Contains information for installing OKD on the target bare-metal host.

      Used with ClusterDeployment to generate the discovery ISO for the managed cluster.

      AgentClusterInstall

      Specifies details of the managed cluster configuration such as networking and the number of control plane nodes. Displays the cluster kubeconfig and credentials when the installation is complete.

      Specifies the managed cluster configuration information and provides status during the installation of the cluster.

      ClusterDeployment

      References the AgentClusterInstall CR to use.

      Used with InfraEnv to generate the discovery ISO for the managed cluster.

      NMStateConfig

      Provides network configuration information such as MAC address to IP mapping, DNS server, default route, and other network settings.

      Sets up a static IP address for the managed cluster’s Kube API server.

      Agent

      Contains hardware information about the target bare-metal host.

      Created automatically on the hub when the target machine’s discovery image boots.

      ManagedCluster

      When a cluster is managed by the hub, it must be imported and known. This Kubernetes object provides that interface.

      The hub uses this resource to manage and show the status of managed clusters.

      KlusterletAddonConfig

      Contains the list of services provided by the hub to be deployed to the ManagedCluster resource.

      Tells the hub which addon services to deploy to the ManagedCluster resource.

      Namespace

      Logical space for ManagedCluster resources existing on the hub. Unique per site.

      Propagates resources to the ManagedCluster.

      Secret

      Two CRs are created: BMC Secret and Image Pull Secret.

      • Image Pull Secret contains authentication information for the OKD image installed on the target bare-metal host.

      ClusterImageSet

      Passed into resources to provide OKD images.