Continuous Integration

    Running kOps in a CI environment can also be useful for upgrading kOps. Simply download a newer version in the CI environment and run a new pipeline. This will allow any changes to be reviewed prior to being applied. This strategy can be extended to sequentially upgrade kOps on multiple clusters, allowing changes to be tested on non-production environments first.

    If you have a solution for a different CI platform or deployment strategy, feel free to open a Pull Request!

    • The GitLab runners that run the jobs need the appropriate permissions to invoke the kOps commands. For clusters in AWS this means providing AWS IAM credentials either with IAM User Keys set as secret variables in the project, or having the runner run on an EC2 instance with an instance profile attached.
    1. A cluster administrator makes a change to a cluster manifest, commits and pushes to a feature branch on GitLab and opens a Merge Request
    2. A reviewer reviews the change to confirm it is as intended, and approves or merges the MR
    3. A “master” pipeline is triggered from this merge commit which runs a kops update cluster.
    4. Once the cluster is updated, is ran which can be used to confirm any nodes that need replacement. The administrator then starts the roll job which runs kops rolling-update cluster --yes and replaces any nodes as necessary.
    • The only: field in each job will need to be updated to reflect the real project’s namespace and name. The two variables will also need to be set to real values.
    • The jobs that make actual changes to the clusters are manually invoked (when: manual) though this could easily be removed to make them automatic.
    • This pipeline setup will create and update existing clusters in place. It does not perform a “blue/green” deployment of multiple clusters.

    In this case, it is possible to use to manage one YAML template and per-cluster values files with which to render the template. See the documentation for more information. kops toolbox template would then be ran before kops replace.

    • This pipeline does not have a true “dryrun” job that can be ran on non-master branches, for example before a merge request is merged. This is because the required kops replace before the kops update cluster will update the live assets in the state store which could impact newly launched nodes that download these assets. PR #6465 could add support for copying the state store to a local filesystem prior to kops replace, allowing the dryrun pipeline to be completely isolated from the live state store.