Release process

    We maintain a branch for kOps 1.21.X, release-1.22 for kOps 1.22.X, etc.

    master is where development happens. We create new branches from master as we prepare for a new minor release. As we are preparing for a new Kubernetes release, we will try to advance the master branch to focus on the new functionality and cherry-pick back to the release branches only as needed.

    Generally we don’t encourage users to run older kOps versions, or older branches, because newer versions of kOps should remain compatible with older versions of Kubernetes.

    Beta and stable releases (excepting the first beta of a new minor version) should be made from the release-1.X branch. Alpha releases may be made on either master or a release branch.

    Typically, kOps alpha releases are created off the master branch and beta and stable releases are created off of release branches. The exception is the first beta release for a new minor version: it is where the release branch for the minor version branches off of master. In order to create the first beta release for a new minor version and a new release branch off of master, perform the following steps:

    1. Update the periodic E2E Prow jobs for the “next” kOps/Kubernetes minor version.
    2. Edit build_jobs.py to add the new minor version to k8s_versions and kops_versions. Also update the list of minor versions in generate_versions(), generate_pipeline(), and generate_presubmits_e2e().
    3. Edit the to add the new minor version to both lists in the file, prefixed with kops-k8s-.
    4. Remove the oldest minor version from each of those lists.
    5. Run the build_jobs.py script.
    6. Create a new milestone in the GitHub repo.
    7. Update prow’s milestone_applier config to update master to use the new milestone and add an entry for the new feature branch. Create this as a separate PR as it will require separate review.
    8. Create the .0-beta.1 release per the instructions in the following section. GitHub Actions will create the release branch when it tags the release.
    9. On the master branch, create a PR to update to the next minor version:
    10. Update OldestSupportedKubernetesVersion and OldestRecommendedKubernetesVersion in
    11. Add a row for the new minor version to upgrade_k8s.md
    12. Fix any tests broken by the now-unsupported versions.
    13. Create release notes for the next minor version. The release notes should mention the Kubernetes support removal and deprecation.

    See for an example.

    Use the hack/set-version script to update versions, using the new version as the argument. Then update the golden tests.

    Commit the changes (without pushing yet):

    1. VERSION=$(tools/get_version.sh | grep VERSION | awk '{print $2}')
    2. git checkout -b release_${VERSION}
    3. git add . && git commit -m "Release ${VERSION}"

    This is the “release commit”. Push and create a PR.

    1. gh pr create -f

    Wait for the PR to merge.

    Reviewing the release commit PR

    To review someone else’s release commit, verify that:

    • A release at that point is desired. (For example, there are no unfixed blocking bugs.)
    • There is nothing in the commit besides version number updates and golden outputs.

    The “verify-versions” CI task will ensure that the versions have been updated in all the expected places.

    Wait for CI job to complete

    It (currently) takes about 30 minutes to run.

    In the meantime, you can compile the release notes…

    Compile release notes

    This step is not necessary for an “.0-alpha.1” release as these are made off of the branch point for the previous minor release.

    The relnotes tool is from .

    For example:

    1. git checkout master
    2. git pull upstream master
    5. FROM=1.21.0-alpha.2 # Replace "1.21.0-alpha.2" with the previous version
    6. DOC=$(expr ${VERSION} : '\([^.]*.[^.]*\)')
    7. git log v${FROM}..v${VERSION} --oneline | grep Merge.pull | grep -v Revert..Merge.pull | cut -f 5 -d ' ' | tac  > /tmp/prs
    8. echo -e "\n## ${FROM} to ${VERSION}\n"  >> docs/releases/${DOC}-NOTES.md
    9. relnotes  -config .shipbot.yaml  < /tmp/prs  >> docs/releases/${DOC}-NOTES.md

    Review then send a PR with the release notes:

    The following tools are prerequisites:

    Create container promotion PR:

    1. cd ${GOPATH}/src/k8s.io/k8s.io
    3. git checkout main
    4. git pull upstream main
    5. git checkout -b kops_images_${VERSION}
    7. cd k8s.gcr.io/images/k8s-staging-kops
    8. echo "" >> images.yaml
    9. echo "# ${VERSION}" >> images.yaml
    10. kpromo cip run --snapshot gcr.io/k8s-staging-kops --snapshot-tag ${VERSION} >> images.yaml

    Currently we send the image and non-image artifact promotion PRs separately.

    1. cd ${GOPATH}/src/k8s.io/k8s.io
    2. git add -p k8s.gcr.io/images/k8s-staging-kops/images.yaml
    3. git commit -m "Promote kOps $VERSION images"

    Create binary promotion PR:

    3. git checkout main
    4. git pull upstream main
    5. git checkout -b kops_artifacts_${VERSION}
    7. rm -rf ./k8s-staging-kops/kops/releases
    8. mkdir -p ./k8s-staging-kops/kops/releases/${VERSION}/
    9. gsutil rsync -r  gs://k8s-staging-kops/kops/releases/${VERSION}/ ./k8s-staging-kops/kops/releases/${VERSION}/
    11. kpromo manifest files --src k8s-staging-kops/kops/releases/ >> artifacts/manifests/k8s-staging-kops/${VERSION}.yaml

    Verify, then send a PR:

    Upon approval and merge of the binary promotion PR, artifacts will be promoted to artifacts.k8s.io via postsubmit. The process is described in detail here.

    Promote to GitHub (all releases)

    Binaries to github (all releases):

    1. cd ${GOPATH}/src/k8s.io/kops/
    2. git checkout v$VERSION
    3. shipbot -tag v${VERSION} -config .shipbot.yaml -src ${GOPATH}/src/k8s.io/k8s.io/k8s-staging-kops/kops/releases/${VERSION}/

    Promote to S3 (stable releases < 1.22)

    1. aws s3 sync --acl public-read ${GOPATH}/src/k8s.io/k8s.io/k8s-staging-kops/kops/releases/${VERSION}/ s3://kubeupv2/kops/${VERSION}/

    Smoke test the release

    This step is only necessary for stable releases (as binary artifacts are not otherwise promoted to artifacts.k8s.io).

    1. wget https://artifacts.k8s.io/binaries/kops/${VERSION}/linux/amd64/kops
    3. mv kops ko
    4. chmod +x ko

    Also run through a kops create cluster flow, ideally verifying that everything is pulling from the new locations.

    • Download release
    • Validate it
    • Add notes
    • Publish it

    Release to Homebrew

    This step is only necessary for stable releases in the latest stable minor version.

    • Following the we must release a compatible homebrew formulae with the release.
    • This should be done at the same time as the release, and we will iterate on how to improve timing of this.

    Update conformance results with CNCF

    This step is only necessary for a first stable minor release (a “.0”).

    Use the following instructions:

    Update latest minor release in documentation

    This step is only necessary for a first stable minor release (a “.0”).

    Create a PR that updates the following documents:

    • Rotate the new version into the version matrix in both and README-ES.md.
    • Remove the “has not been released yet” header in the version’s release notes.

    This step is only necessary for a first beta minor release (a “.0-beta.1”).

    Create a PR that updates the following document:

    • Add a reference to the version’s release notes in

    Update the alpha channel and/or stable channel

    Once we are satisfied the release is sound:

    Once we are satisfied the release is stable:

    • Bump the kOps recommended version in the stable channel