Scorecard

    Tests are implemented within test images that are configured and constructed to be executed by scorecard.

    Scorecard assumes it is being executed with access to a configured Kubernetes cluster. Each test is executed within a Pod by scorecard, from which pod logs are aggregated and test results sent to the console.

    Scorecard has built-in basic and OLM tests, and it also provides a means to execute custom test definitions.

    Requirements

    The scorecard tests make no assumptions as to the state of the operator being tested. Creating operators and custom resources for an operator are left outside the scope of the scorecard itself.

    Scorecard tests can however create whatever resources they require if the tests are designed for resource creation.

    Running the Scorecard

    1. A default set of kustomize files should have been scaffolded by . If that is not the case, run operator-sdk init as you would have to initialize your project and copy scaffolded files:

    The default config generated by this kustomization can be immediately run against your operator. See the config file section for an explanation of the configuration file format.

    1. (Re)enerate manifests and metadata for your Operator. make bundle will automatically add scorecard annotations to your bundle’s metadata, which is used by the scorecard command to run tests.
    2. Execute the scorecard command. See the for an overview of command invocation.

    The scorecard test execution is driven by a configuration file named config.yaml, generated by make bundle. Note that if run make bundle that any changes you have made to config.yaml will be overwritten. To persist any changes to config.yaml you can update the kustomize templates found in the config/scorecard directory. The configuration file is located at the following location within your bundle directory (bundle/ by default):

    1. $ tree ./bundle
    2. ./bundle
    3. ...
    4. └── tests
    5.     └── scorecard
    6.         └── config.yaml

    The following YAML spec is an example of the scorecard configuration file:

    1. kind: Configuration
    2. apiversion: scorecard.operatorframework.io/v1alpha3
    3. metadata:
    4.   name: config
    5. stages:
    6. - parallel: true
    7.   tests:
    8.     entrypoint:
    9.     - scorecard-test
    10.     - basic-check-spec
    11.     labels:
    12.       suite: basic
    13.       test: basic-check-spec-test
    14.   - image: quay.io/operator-framework/scorecard-test:latest
    16.     - scorecard-test
    17.     - olm-bundle-validation
    18.     labels:
    19.       suite: olm
    20.       test: olm-bundle-validation-test

    The configuration file defines the tests that scorecard executes. Tests are grouped into stages for fine-grained control of parallelism. The following fields of the scorecard configuration file define the test as follows:

    Command Args

    The scorecard requires a positional argument that holds either the on-disk path to your operator bundle or the name of a bundle image. Note that the scorecard does not run your operator but merely uses the scorecard configuration within the bundle contents to know which tests to execute.

    For further information about the flags see the CLI documentation.

    Parallelism

    The configuration file allows operator developers to define separate stages for their tests. Stages run sequentially in the order they are defined in the configuration file. A stage contains a list of tests and a configurable parallel setting.

    By default (or when a stage explicitly sets parallel to false), tests in a stage are run sequentially in the order they are defined in the configuration file. Running tests one at a time is helpful to guarantee that no two tests interact and conflict with each other.

    However, if tests are designed to be fully isolated, they can be parallelized. To run a set of isolated tests in parallel, include them in the same stage and set parallel to true. All tests in a parallel stage are executed simultaneously, and scorecard waits for all of them to finish before proceding to the next stage. This can make your tests run much faster.

    Selecting Tests

    Tests are selected by setting the --selector CLI flag to a set of label strings. If a selector flag is not supplied, then all the tests within the scorecard configuration file are executed.

    Tests are executed serially, one after the other, with test results being aggregated by scorecard and written to stdout.

    To select a single test (basic-check-spec-test) you would enter the following:

    1. $ operator-sdk scorecard <bundle_dir_or_image> -o text --selector=test=basic-check-spec-test

    To select a suite of tests, olm in this case, you would specify a label that is used by all the OLM tests:

    1. $ operator-sdk scorecard <bundle_dir_or_image> -o text --selector=suite=olm

    The scorecard ships with pre-defined tests that are arranged into suites.

    OLM Test Suite

    Scorecard Output

    The --output flag specifies the scorecard results output format.

    See an example of the JSON format produced by a scorecard test:

    1. {
    2.   "apiVersion": "scorecard.operatorframework.io/v1alpha3",
    3.   "kind": "TestList",
    4.   "items": [
    5.     {
    6.       "kind": "Test",
    7.       "apiVersion": "scorecard.operatorframework.io/v1alpha3",
    8.       "spec": {
    9.         "image": "quay.io/operator-framework/scorecard-test:latest",
    10.         "entrypoint": [
    11.           "scorecard-test",
    12.           "olm-bundle-validation"
    13.         ],
    15.           "suite": "olm",
    16.           "test": "olm-bundle-validation-test"
    17.         }
    18.       "status": {
    19.         "results": [
    20.           {
    21.             "name": "olm-bundle-validation",
    22.             "log": "time=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Found metadata directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=debug msg=\"Getting mediaType info from manifests directory\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Found annotations file\" name=bundle-test\ntime=\"2020-06-10T19:02:49Z\" level=info msg=\"Could not find optional dependencies file\" name=bundle-test\n",
    23.             "state": "pass"
    24.           }
    25.         ]
    26.       }
    27.     }
    28.   ]
    29. }

    Text format

    See an example of the text format produced by a scorecard test:

    1. --------------------------------------------------------------------------------
    2. Image:      quay.io/operator-framework/scorecard-test:latest
    3. Entrypoint: [scorecard-test olm-bundle-validation]
    4. Labels:
    5.     "suite":"olm"
    6.     "test":"olm-bundle-validation-test"
    7. Results:
    8.     Name: olm-bundle-validation
    9.     State: pass
    10.     Log:
    11.         time="2020-07-15T03:19:02Z" level=debug msg="Found manifests directory" name=bundle-test
    12.         time="2020-07-15T03:19:02Z" level=debug msg="Found metadata directory" name=bundle-test
    13.         time="2020-07-15T03:19:02Z" level=debug msg="Getting mediaType info from manifests directory" name=bundle-test
    14.         time="2020-07-15T03:19:02Z" level=info msg="Found annotations file" name=bundle-test
    15.         time="2020-07-15T03:19:02Z" level=info msg="Could not find optional dependencies file" name=bundle-test

    NOTE The output format spec for each test matches the Test type layout.

    Exit Status

    The scorecard return code is 1 if any of the tests executed did not pass and 0 if all selected tests pass.

    Scorecard will execute custom tests if they follow these mandated conventions:

    • tests are implemented within a container image
    • tests accept an entrypoint which include a command and arguments
    • tests produce v1alpha3 scorecard output in JSON format with no extraneous logging in the test output
    • tests can obtain the bundle contents at a shared mount point of /bundle
    • tests can access the Kubernetes API using an in-cluster client connection

    See here for an example of a custom test image written in Go.

    Writing custom tests in other programming languages is possible if the test image follows the above guidelines.