Breaking changes

Significant changes

kops now supports running with objects as CRDs, stored in a kubernetes apiserver.
The apiGroup for kops objects has changed from kops to , to support CRDs. You can continue to provide either apiGroup as input (but you should ideally move to kops.k8s.io), but the output will always be of the kops.k8s.io form.
Rolling updates are much faster by default. A lot of the time-padding that was in previous versions has been replaced with reliance on validation. The --cloudonly case is much faster than previously, which we believe to be correct because we expect this is normally for disaster-recovery scenarios, but you may want to specify longer timings via flags if you are relying on time-based delays.

Required Actions

Full change list since 1.14.0 release

kops 1.14.0-beta.2 to 1.15.0-alpha.1

Release 1.14.0-alpha.1 @justinsb
Put 1.13 and 1.14 into channels @justinsb
1.12 release notes: populate list of PRs @justinsb
Carry Provisioned IOPS to Terraform and CloudFormation templates @mmailhos
pin nvidia-docker2 version to avoid installation failure @adrianlyjak
pkg/model: Fix dropped error @alrs
Using const() defines constants together (part:1) @alrs, #6789
Using const() defines constants together (part:3) #6791
Update rules go #6766
Update etcd3-migration.md doc ,@mikesplain
KubeAPIServer HTTP2 Stream Parameter @gambol99
Refactor names of URLs in assets to clarify their purpose @justinsb
Update docker README.md file, delete @xichengliudui
Switch to golang 1.11.5 @justinsb
Switch to golang 1.12.1 @justinsb
Using const() defines constants together (part:3) @xichengliudui
Using const() defines constants together (part:4) @xichengliudui
[docs] Use env var for state store examples @elithrar
Update README.md @wangxy518
set kubernetes version to 1.12.8 to match current release @chrisz100
Canal manifest updates for k8s v1.12+ @KashifSaadat
Update readme chart given alphas and betas @mikesplain
Fix typo in aws-iam-authenticator image field name @rifelpet
Remove verify bazel and expose error @mikesplain
Add t3a family @mikesplain
Add support for AWS ap-east-1 region @wxdao
update tolerations to openstack external cloud provider @zetaab
[Unit Tests] Add unit tests for create_kubecfg file @srikiz
Protect against nil derefence @justinsb
Support Scale from 0 with Lauch Templates @granular-ryanbonham
Remove spurious cadvisor dependency @justinsb
makefile: add gazelle alias for bazel-gazelle @justinsb
bazel: fix distroless imports for latest bazel @justinsb
Update kubernetes dependencies to k8s 1.13.5 @justinsb
Switch from glog to klog @justinsb
travis: Remove go-vet and boilerplate checking @justinsb
Use existing SSHKeyName if no public key is created. @rralcala
Start CRDification: Change apigroup to kops.k8s.io @justinsb
Simply bazel test using exclude pattern @justinsb
Include aws-cloud-provider roles in 1.15 @justinsb
Fix machine types with klog @mikesplain
Avoid concurrent write corruption to /etc/hosts @justinsb
Add i3en instance types @mikesplain
Fix typo in docker healthcheck @TristanPeers
Update to etcd-manager 1.0.20190509 @justinsb
Call klog.InitFlags in dns-controller @justinsb
Use klog logging from 1.15 @justinsb
S3 VFS: Default to current region from metadata service @justinsb
Canal v3.7.2 for k8s v1.12+ @KashifSaadat
Subnet Update Consistency @drekle
Configure AMIs for 1.12 @justinsb
Fix Docker not being installed on Ubuntu 16.04 @mfrister
bumped k8s 1.11 versions to 1.11.10 in alpha channel @idealhack
Issue #6945 @pkutishch
Generate CRDs for kops API types @justinsb
etcd-manager: Update to 3.0.20190513 @justinsb
add node-exporter to allowed ports @zetaab
Make gofmt fails find usage @drekle
Update commitlog relnotes for 1.12.0 @justinsb
1.12 highlight changelog @granular-ryanbonham
Mention version of kOps that introduced new features @rifelpet
Terraform: fix options field, should be spot_options @kimxogus
Add shortNames and columns to InstanceGroup CRD @justinsb
Add script to verify CRD generation @justinsb
Update README.md to reflect 1.12 release @natebwangsut
add kops instancegroup tag to metadata @zetaab
Don’t panic when deleting instancegroups @justinsb
Support using kops CLI with CRDs @justinsb
etcd-manager: update to 3.0.20190516 @justinsb
VPC cleanup: recognize the error code for concurrent VPC deletion @justinsb
Recommend kops 1.12.1 @justinsb
Add relnotes list for 1.12.1 @justinsb
Makefile: keep go vet simple @justinsb
Update go_version to 1.12.5 @justinsb
Start relnotes for 1.13 and 1.14 @justinsb
Speed up rolling-update - longer timeout on validation, less scheduled holds @justinsb
Update stretch dependencies and kubeup @mikesplain
Bump alpha-channel of k8s @justinsb
Fix machine empheral disks @mikesplain
Add docs for cpuCFSQuota / cpuCFSQuotaPeriod @thomaspeitz
implement append admission controllers @zetaab
Add documentation for etcd-manager backup/restore procedures @dzoeteman
Fix typo on node-authorizer prometheus metric @KashifSaadat
Openstack delete dynamic floating ip in delete cluster @drekle
Updated docs for openstack cloud provider. @prankul88
Change versions to fix memory.limit_in_bytes: device or resource busy @flouthoc
K8s 1.12.8 to stable 1.12.9 to alpha @granular-ryanbonham
Fix link to Calico route reflectors documentation + typo @Misdre
Mark ENI 0 as delete_on_termination for LaunchTemplates @granular-ryanbonham
Add Debian 10 (buster) support @zetaab
Openstack support for rolling-update status @drekle
Upgrade AWS VPC CNI provider to 1.5.0 @rifelpet
Documentation Cleanup @rifelpet
Adding affinity and PDB to dns. @michalschott
bumped k8s 1.11 versions to 1.11.10 in stable channel @idealhack
Add support for SpotPrice and Mixed Instance ASGs @rifelpet
support apiserver admission-control-config-file flag @r0fls
typo fix: fix kops-server-push -> make kops-server-push @Sn0rt
Add rdrgmnzs to the approvers list in OWNERS @rdrgmnzs
Flatcar support @mazzy89
Don’t precreate etcd DNS records if we’re using etcd-manager @rifelpet
Update Docs for Calico Backend for kops 1.12 @gjtempleton
Update Canal to v3.7.3 @KashifSaadat
Improve docs on labels @granular-ryanbonham
Allow user to set the —kube-api-qps and —kube-api-burst flags on KubeControllerManager @rdrgmnzs
Egress proxy for etcd manager @austinmoore-
[Unit Tests] Added unit test for kube proxy builder @srikiz
add c5.12xlarge, c5.24xlarge, c5.metal, i3en.metal @rekcah78
Simplify go test command @justinsb
Spotinst: New instance group type: Ocean @liranp
Fix the link to the Prow commands. @cjwagner
add masterPublicName support in kops set cluster @camilosantana
Update aws-iam-authenticator image to 0.4.0 @rifelpet
Added some changes to openstack.md file @prankul88
Relnotes for 1.12.2 @justinsb
Add updated 1.12 image to the alpha channel @justinsb
Instance protection @mikesplain
Remove kube-proxy resource-container flag @justinsb
Drop missing sources when building utils image @KashifSaadat
goimports update @KashifSaadat
Add more debug info for when cluster path doesnt match @jayunit100
Canal v3.7.4 @KashifSaadat
Upgrade Calico to 3.7.2 @asincu
Spotinst: Ocean’s Strategy object is optional @liranp
update instances list with make update-machine-types @rekcah78
Possibility to use OpenStack without lbaas (loadbalancer) @zetaab
Clear append admission plugins before inserting flags to kube-apiserver @zetaab
Use NodeAuthorizer config options instead of soely hard-codes @jacksontj
doc: support to debug kops-apiserver @Sn0rt
GCE tutorial markdown formatting @flaviamissi
Make an actual deep-copy of the state @jacksontj
Set priority for static pods @vainu-arto
Allow setting Limit & Request for aws-iam-authenticator @rdrgmnzs
Delete the function keyword to prevent shellcheck from failing @xichengliudui
Bumping calico to 3.7.4. @michalschott
Update metrics server image @elisiano
Use readinessProbe for weave-net instead of livenessProbe @ReillyProcentive
Add some permissions to cluster-autoscaler clusterrole @Coolknight
Spotinst: Rolling update always reports NeedsUpdate @liranp
Add documentation example for running kOps in a CI environment @rifelpet
Calico -> 3.7.4 for older versions @justinsb
[Issue-7148] Legacyetcd support for Digital Ocean @srikiz
Stop .gitignoring all files named go-bindata @justinsb
Create hack/update-expected.sh to update test output @justinsb
replace behavior for @aws hostnameOverride @jacksontj
Rhel8 support @cassandracomar
Update DigitalOcean CCM to v0.1.16 @timoreimann
Replace use of cmdutil IsFilenameSliceEmpty @justinsb
GCE: support ipalias networking mode, named “gce” @justinsb
Move NTP and misc packages initialization to code @justinsb
Machine types fix @mikesplain
Improve channel updates @granular-ryanbonham, #7133
Rationalize golden-output comparison #7290
hack/update-expected: regenerate gobindata #7311
Add me as reviewer #7313
Update Calico to v3.8.0 #7257
Manifest hashing: move trimming out of hash function #7312
Adding documentation to mitigate workload outages on AWS #7292
Mount FlexVolume directory in kube-controller-manager pod #6874
remove code: remove kops-server chart #7324
Bump alpha channel with latest kubernetes versions #7338
Default etcd-version to 3.3.10 for >= 1.14 #7341
Warn/prevent if the version of etcd is unsupported with etcd-manager #7340
Update Image version and RBAC for Citrix Ingress Controller #7335
Promote k8s 1.12.9 from alpha -> stable #7337
Update repo-infra and distroless for bazel fixes #7348
Cross-Zone Load Balancing for API ELB #6958
stop kubelet to prevent orphan containers #7345
Update default flexvolumepath for COS #7339
Promote 1.12 image from alpha to stable #7343
Update kube-router to 0.3.1 #7317
Enable scraping of weave metrics #7326
Bump etcd-manager to 3.0.20190801 #7349
Add mappings for Webhook authorization mode. #7344
Set and mount the correct volume plugin dir based on OS #7355
Don’t default adding MIMEBOUNDARY headers when a mixed instances policy is set #7370
Add release notes for 1.13.0 beta.2 -> 1.13.0 #7372
Release notes for 1.12.2 -> 1.12.3 #7373
correct typo in output message #7380
Remove extraneous note in 1.13 release notes #7374
protokube/gce_volume.go: error info correction #7382
cleanup: client.go error message words correction #7394
awstasks: fix misspelled words in logging. #7412
AWS SDK v1.23.0 #7404
Update Compatibility Table in Readme #7408
Add maxPersistentVolumes to support the KUBE_MAX_PD_VOLS scheduler setting #7346
add zetaab as reviewer #7403
Support mirroring for nodeup also #7377
Use Cluster Proportional Autoscaler for CoreDNS 1.12+ #7400
Upgrading k8s-srcdst to v0.2.2 #7388
add OpenStack etcd-manager support #7395
Bump k8s versions in alpha channel #7422
Update AMIs in alpha channels #7420
skip verification when the file already installed #7387
cleanup: fix error message typos #7425
fix Typo ‘the the’ -> ‘the’ #7441
ali.go: cleanup error words in messages #7433
promote k8s versions #7437
Dont set ExperimentalCriticalPodAnnotation feature gate in k8s 1.16 #7430
fix typo “specifiction” -> “specification” #7440
Add relnotes for 1.14.0-beta.1 #7443
bash script: don’t assume nodeup filename #7448
Add doc for using custom CA #7434
nodeup download should try all mirrors #7447
Add nodeup to shipbot targets for release upload #7449
Update channel recommended versions for kops versions #7446
fix typo “in ingards to” -> “in regard to” #7451
Update weave to 2.5.2 #7444
cloudformation tests: use standard file comparison #7450
Look for sha256 and sha1 files for artifacts #7468
[Digital Ocean] DO-7442 upgrade godo client to latest version #7467
Fix Flatcar distro #7464
fix typo “new” -> “newer” #7462
print all failure messages #7465
[DO-7148] Digital Ocean support for etcd-manager #7435
Publish sha256 artifacts for kops itself #7471
Ignore empty hashfiles #7472
Update to kubernetes 1.15 #7470
util/pkg/vfs: Fix swallowed errors #7483
Set GOPROXY in travis builds #7485
Allow configure ip to ip mode in calico #7481
Add exec-opts options to dockerconfig #7460
move OpenStack from alpha to beta #7488
Add support for netExtraArgs #7429
Cleanup versions, deprecate kops 1.10, remove k8s 1.8 #7491
Create verify-gomod script #7498
Update to golang 1.12.9 #7499
Corrected spelling of ‘we’ in the documentation #7495
fix-up docs/releases/1.11-NOTE.md spelling mistake #7501
Copy well-known users from apiserver #7497
Replace resource.FilenameOptions with []string #7500
Configure calico MTU #7480
Create env-var helper function #7505
Label AWS ASGs with kops.k8s.io/instancegroup #7504
Support for using hostPort when using flannel #7295
Remove unused ClientGetter from Drain code #7509
DeleteLocalData on drain #7510
Updating the vendored gazelle to match workspace. #7511
Add verbosity #7514
fix(addons/coredns.addons.k8s.io) Workaound to stop coredns crashing on 1.3.1 version #7492
Update rules_docker with python2 workaround #7508
Update and add back some sizes #7515
Don’t try to delete ElasticIPs of NatGateway is shared #7525
fix(addons/networking.projectcalico.org) calico kube-controllers is needed in CRD mode #7517
remove default insecure from openstack #7524
docs: fix link to Metrics Server user guide #7479
fix static check error in vfssync.go #7482
fix(upup/models/cloudup/resources/addons/coredns.addons.k8s.io) missing resourceVersion #7477
modify-doc-small-mistake #7519

Calico update and typha ,@mikesplain
[Feature] CoreDNS: External CoreFile option @gjtempleton
Fix gomod errors @mikesplain
Log more sensibly when we can’t get sha256 @justinsb
Add horizontalPodAutoscalerDownscaleStabilization @mikesplain
Fix kops for us-gov-east-1 #7564 @ibrf
Fix Dropped Errors in upup @alrs
add cilium in error message @PascalBourdier
[DO-7442] Digital Ocean add consistent volume and droplet tags for multi master feature @srikiz
Expose API Server flags needed for AWS pod identities @rifelpet
Add logrotate for etcd/etcd-events.log @mikesplain
Updated container-selinux url to point to the right path @igarcia-sugarcrm, #7609
Check the HTTP response code when downloading URLs #7611
Clean security groups if api/ssh ips are removed from config #7561
Skip Docker install #6957
Add —wait argument to kops validate #7371
Fixed “NeedsUpdate” status of nodes in mixedinstancegroups after rolling update #7445
Associate subnets to port within OpenStack #7578
fix instance name #7641
Use without external router (OpenStack) #7644
Updating master IAM policies. #7580
Cherrypick #7581 into release 1.15 #7671
Pull centos.org packages from the vault #7674
Align AWS and kops validation for spot allocation strategy #7660
Limit calico cpu request to 100m #7688
Cherrypick #7690 onto release 1.15 #7693
Update etcd-manager with OpenStack fixes #7710
Change Cilium templates to standalone version ,@olemarkus
Update DigitalOcean CCM to v0.1.20 @timoreimann
Cilium standalone continuation @olemarkus
Add calico 3.9.1 @mikesplain
Fix some bugs reported by staticcheck @rifelpet
Add arg min-port=1024 to dnsmasq container in kube-dns @nr17
Add artifacts.k8s.io to mirror list @justinsb
Upgrade Amazon VPC CNI plugin to 1.5.4 @rifelpet
Add event ttl flag @tioxy
Kubelet configuration: Maximum pods flag is miscalculated when using Amazon VPC CNI @liranp
fix(apiserver): allow multiple service-account-key-file @hatappi
Openstack: value if spec does not associate public ips @mitch000001

1.15.0-beta.1 to 1.15.0

Cherry-pick #7807 to release-1.15 #7809
allow protocol rules in master #7835
Revert “Upgrade Amazon VPC CNI plugin to 1.5.4” #7847
Add back calico metrics options #7885
Remove extraneous document separator causing failures applying addons #7857
add missing priorityClassName to flannel DaemonSet #7842
Create PodDisruptionBudget for kube-dns in kube-system namespace ,@justinsb
Machine types updates @mikesplain
Add support for newer Docker versions @hakman

Add indent template function and use it to fix KubeDNS.ExternalCoreFile rendering @rochacon
fix(openstack): fix additional security groups on instance groups @mitch000001
Fix Handling of LaunchTemplate Versions for MixedInstancePolicy @granular-ryanbonham
Fix mounting Calico “flexvol-driver-host” in CoreOS @hakman
Complete support for Flatcar @mazzy89
Openstack: Fix cluster floating ips @mitch000001
Bump cilium version to 1.6.4 @olemarkus
mark weavenet-pod as system-critical @jochen42
cilium: don’t try to mount sys/fs/bpf if already mounted @justinsb
Update copyrights for 2020 @hakman
Fix rendering of the Node Authorizer template @KashifSaadat
Cherry pick #7874 onto 1.15 @k8s-ci-robot
Backport the k8s 1.9 required action release note @johngmyers
Don’t output empty sections in the manifests @justinsb, #8317
Fix issues with older versions of k8s for basic clusters ,@rifelpet
CoreDNS default image bump to 1.6.6 to resolve CVE @gjtempleton
Don’t load nonexistent calico-client cert when CNI is Cilium @johngmyers
kOps releases - prefix git tags with v @rifelpet

1.15.1 to 1.15.2

Fix Github download url for nodeup ,@justinsb
GCS: Don’t try to set ACLs if bucket-policy only is set @justinsb
Cilium - Add missing Identity Allocation Mode to Operator Template @daviddyball
Make it possible to enable Prometheus metrics for Cilium @olemarkus

Stabilize sequence of “export xx=xxx” statements @mitch000001
Properly detect that bpffs has been mounted @olemarkus
Update to etcd-manager 3.0.20200428 @justinsb