Validating Operators using the scorecard tool

    • Validate that your Operator project is free of syntax errors and packaged correctly

    • Review suggestions about ways you can improve your Operator

    While the Operator SDK subcommand can validate local bundle directories and remote bundle images for content and structure, you can use the scorecard command to run tests on your Operator based on a configuration file and test images. These tests are implemented within test images that are configured and constructed to be executed by the scorecard.

    The scorecard assumes it is run with access to a configured Kubernetes cluster, such as OKD. The scorecard runs each test within a pod, from which pod logs are aggregated and test results are sent to the console. The scorecard has built-in basic and Operator Lifecycle Manager (OLM) tests and also provides a means to execute custom test definitions.

    Scorecard workflow

    1. Create all resources required by any related custom resources (CRs) and the Operator

    2. Create a proxy container in the deployment of the Operator to record calls to the API server and run tests

    3. Examine parameters in the CRs

    The scorecard tests make no assumptions as to the state of the Operator being tested. Creating Operators and CRs for an Operators are beyond the scope of the scorecard itself. Scorecard tests can, however, create whatever resources they require if the tests are designed for resource creation.

    scorecard command syntax

    The scorecard requires a positional argument for either the on-disk path to your Operator bundle or the name of a bundle image.

    For further information about the flags, run:

    1. $ operator-sdk scorecard -h

    Scorecard configuration

    The scorecard tool uses a configuration that allows you to configure internal plugins, as well as several global configuration options. Tests are driven by a configuration file named config.yaml, which is generated by the make bundle command, located in your bundle/ directory:

    1. ./bundle
    2. ...
    3. └── tests
    4.     └── scorecard
    5.         └── config.yaml

    Example scorecard configuration file

    1. kind: Configuration
    2. apiversion: scorecard.operatorframework.io/v1alpha3
    3. metadata:
    4.   name: config
    5. stages:
    6. - parallel: true
    7.   tests:
    8.   - image: quay.io/operator-framework/scorecard-test:v1.28.0
    9.     entrypoint:
    10.     - scorecard-test
    11.     - basic-check-spec
    12.     labels:
    13.       suite: basic
    14.       test: basic-check-spec-test
    15.   - image: quay.io/operator-framework/scorecard-test:v1.28.0
    16.     entrypoint:
    17.     - scorecard-test
    18.     - olm-bundle-validation
    19.     labels:
    20.       suite: olm
    21.       test: olm-bundle-validation-test

    The configuration file defines each test that scorecard can execute. The following fields of the scorecard configuration file define the test as follows:

    The scorecard ships with pre-defined tests that are arranged into suites: the basic test suite and the Operator Lifecycle Manager (OLM) suite.

    Running the scorecard tool

    A default set of Kustomize files are generated by the Operator SDK after running the init command. The default bundle/tests/scorecard/config.yaml file that is generated can be immediately used to run the scorecard tool against your Operator, or you can modify this file to your test specifications.

    Prerequisites

    • Operator project generated by using the Operator SDK

    Procedure

    1. Run the scorecard against the on-disk path to your Operator bundle or the name of a bundle image:

      1. $ operator-sdk scorecard <bundle_dir_or_image>

    The --output flag for the scorecard command specifies the scorecard results output format: either text or json.

    1. {
    2.   "apiVersion": "scorecard.operatorframework.io/v1alpha3",
    3.   "kind": "TestList",
    4.   "items": [
    5.     {
    6.       "kind": "Test",
    7.       "apiVersion": "scorecard.operatorframework.io/v1alpha3",
    8.       "spec": {
    9.         "image": "quay.io/operator-framework/scorecard-test:v1.28.0",
    10.         "entrypoint": [
    11.           "scorecard-test",
    12.           "olm-bundle-validation"
    13.         ],
    14.         "labels": {
    15.           "suite": "olm",
    16.           "test": "olm-bundle-validation-test"
    17.         }
    18.       },
    19.       "status": {
    20.         "results": [
    21.           {
    22.             "name": "olm-bundle-validation",
    23.             "log": "time=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n",
    24.             "state": "pass"
    25.           }
    26.         ]
    27.       }
    28.     }
    29.   ]
    30. }

    Example text output snippet

    1. --------------------------------------------------------------------------------
    2. Image:      quay.io/operator-framework/scorecard-test:v1.28.0
    3. Entrypoint: [scorecard-test olm-bundle-validation]
    4. Labels:
    5.     "suite":"olm"
    6.     "test":"olm-bundle-validation-test"
    8.     Name: olm-bundle-validation
    9.     State: pass
    10.         time="2020-07-15T03:19:02Z" level=debug msg="Found manifests directory" name=bundle-test
    11.         time="2020-07-15T03:19:02Z" level=debug msg="Found metadata directory" name=bundle-test
    12.         time="2020-07-15T03:19:02Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test
    13.         time="2020-07-15T03:19:02Z" level=info msg="Found annotations file" name=bundle-test
    14.         time="2020-07-15T03:19:02Z" level=info msg="Could not find optional dependencies file" name=bundle-test

    Selecting tests

    Scorecard tests are selected by setting the --selector CLI flag to a set of label strings. If a selector flag is not supplied, then all the tests within the scorecard configuration file are run.

    Tests are run serially with test results being aggregated by the scorecard and written to standard output, or stdout.

    Procedure

    1. To select a single test, for example basic-check-spec-test, specify the test by using the --selector flag:

    2. To select a suite of tests, for example olm, specify a label that is used by all of the OLM tests:

      1. $ operator-sdk scorecard <bundle_dir_or_image> \
      2.     -o text \
      3.     --selector=suite=olm

    3. To select multiple tests, specify the test names by using the selector flag using the following syntax:

      1. $ operator-sdk scorecard <bundle_dir_or_image> \
      2.     -o text \
      3.     --selector='test in (basic-check-spec-test,olm-bundle-validation-test)'

    As an Operator author, you can define separate stages for your tests using the scorecard configuration file. Stages run sequentially in the order they are defined in the configuration file. A stage contains a list of tests and a configurable parallel setting.

    By default, or when a stage explicitly sets parallel to false, tests in a stage are run sequentially in the order they are defined in the configuration file. Running tests one at a time is helpful to guarantee that no two tests interact and conflict with each other.

    However, if tests are designed to be fully isolated, they can be parallelized.

    Procedure

    • To run a set of isolated tests in parallel, include them in the same stage and set parallel to true:

      1. apiVersion: scorecard.operatorframework.io/v1alpha3
      2. kind: Configuration
      3. metadata:
      4.   name: config
      5. stages:
      6. - parallel: true (1)
      7.   tests:
      8.   - entrypoint:
      9.     - scorecard-test
      10.     - basic-check-spec
      11.     image: quay.io/operator-framework/scorecard-test:v1.28.0
      12.     labels:
      13.       suite: basic
      14.       test: basic-check-spec-test
      15.   - entrypoint:
      16.     - scorecard-test
      17.     - olm-bundle-validation
      18.     image: quay.io/operator-framework/scorecard-test:v1.28.0
      19.     labels:
      20.       suite: olm
      21.       test: olm-bundle-validation-test

      All tests in a parallel stage are executed simultaneously, and scorecard waits for all of them to finish before proceding to the next stage. This can make your tests run much faster.

    Custom scorecard tests

    The scorecard tool can run custom tests that follow these mandated conventions:

    • Tests are implemented within a container image

    • Tests accept an entrypoint which include a command and arguments

    • Tests produce v1alpha3 scorecard output in JSON format with no extraneous logging in the test output

    • Tests can obtain the bundle contents at a shared mount point of

    • Tests can access the Kubernetes API using an in-cluster client connection

    Writing custom tests in other programming languages is possible if the test image follows the above guidelines.

    The following example shows of a custom test image written in Go:

    Example custom scorecard test