Significant changes

Components are no longer allowed to interact with etcd directly. Calico will be switched to use CRDs instead of directly with etcd. This is a disruptive upgrade, please read the calico notes in the

Required Actions

Please back-up important data before upgrading, as the is higher risk than most upgrades. The upgrade is disruptive to the masters, see notes above.
Note that the upgrade for Calico users is disruptive, because it requires switching from direct-etcd-storage to CRD backed storage.

Full change list since 1.11.0 release

1.11.0 to 1.12.0-alpha.1

machine-type generator: Warn if instance type not in ENI map @justinsb
Include name of unhealthy component in validation error @justinsb
Bump alpha channel kubernetes versions @justinsb
Add missing locking to awsmock LaunchConfigurations @justinsb
Add a1 and c5n instance types @justinsb
Simplify makefile for update-machine-types @justinsb
Update docs, removing brew —devel @mikesplain
machine-types: remove duplicate dedup @justinsb
Update amazon cni to 1.3.0 @mikesplain
Enable HPA tolerance configuration @rlees85
Update addons dashboard version @jeefy
Spotinst: Bump controller image @liranp
Add cni to usage network option for kops create cluster @nak3
Workspace updates for bazel / fix tests @mikesplain
Promote alpha channels to stable @mikesplain
Add GCE europe-north1-{a,b,c} @eetujalonen
Add self to security contacts @mikesplain
Fix missed stable channel upgrade path @mikesplain
Fix Calico upgrade job to use the correct version @tmjd
Fix for when node and master use the same SG. @rdrgmnzs
Add experimental and metrics flags for docker @rbtcollins
Add y flag for upgrade command for consistency @mikesplain
Add-ons spec example is missing “manifest”. @qlikcoe
ExperimentalAllowedUnsafeSysctls has moved to AllowedUnsafeSysctls in k8s 1.11 @rdrgmnzs
Let a user specify the validation timeout when rotating a cluster. @rdrgmnzs
fix(docs): fix the compatibility matrics on hpa.md @Cryptophobia
bump prometheus-operator version and deploy file @zouyee
update heapster version and mark it retired @zouyee
Add Docker 18.06.1 for CentOS and RHEL 7 @bcorijn
Print —name with @joshbranham
Add —post-drain-delay to rolling-update cluster command @rifelpet
Adding kubernetes-dashboard v1.10.1 deployment to kops addons @schweizerbolzonello
Consider pending pods to be a validation failure @justinsb
Adding support for the new Stockholm region @liranp
Document how to update an existing vendored dependency @justinsb
Update to k8s 1.12 libraries @rdrgmnzs
Bump channels and bump alpha to latest @mikesplain
Automagically use curl instead of wget if that’s what’s available @eherot
cloudmock: replace unimplemented methods with interface embedding @justinsb
bazel: cleanup gobindata generation @justinsb
Update apimachinery for k8s 1.12 @justinsb
Bulk spelling fixes @justinsb
Don’t panic when an etcd cluster is added @justinsb
Update aws-sdk-go to 1.16.9 @justinsb
Add p3dn.24xlarge @mikesplain
Rationalize deserialiation code @justinsb
Always log when a retry loop fails @justinsb
Update compatibility for v1.11.0 @mikesplain
AWS SDK v1.16.11 @gambol99
nodeup: include underlying error in error message @andrestc
release process: add the relnotes command @justinsb
Fix missed error check in hasPlaceHolderIP @justinsb
Create dev-upload tasks, for a faster upload during dev builds @justinsb
Update recommended kubernetes version @justinsb
Release notes for 1.11 @justinsb
fixed the sentence mistake @abhijitio
update calico version to version 3.4.0 @chrisz100
Remove duplicate Deployment for prometheus-operator @Smirl
Update aws-china.md @qqshfox
Recognize 2019 as a year @justinsb
Change jessie to stretch @abhijitio
Included type in SSL certificate documentation @walkafwalka
Update distroless @justinsb
Promote alpha kubernetes versions to stable @justinsb
Create prow-postsubmit target for release candidates @justinsb
Include windows build in distribution @justinsb
Fix kubelet api admin @gambol99
GCE terraform: map source tags in firewallrule @justinsb
GCE terraform: support labels @justinsb
Add extra privilege to prometheus-k8s ClusterRole #6305
Kubelet API RBAC Manifest #6317
Upgrading coredns version to 1.3.0 #6326
Release 1.12.0-alpha.1 #6257
Retry Logging #6327
Fix prow-postsubmit by copying prebuilt archive in bazel #6328
Remove Initializers from default admission plugins for 1.12+ #6350
include docker 18.06.1 missed dependency #6338
Fix alternative AWS partitions in custom instance profiles #6226
Add doc regarding upgrading to CoreDNS #6344
AWS: Enable ICMP Type 3 Code 4 for API server ELBs #6297
Additional Storage & Volume Mounting #6066
kOps for Openstack ,@drekle,,@marsavela
Update go version to 1.10.8 @justinsb
Suffix openstack subnet name with cluster name @wozniakjan
minor grammar improvements to kops terraform docs @discdiver
Docs: Drop last DrainAndValidateRollingUpdate note @meeee
Allow users to set kubelet cpu-cfs-quota and cpu-cfs-quota-period flags @wndhydrnt
implement etcd status for openstack @zetaab
remove using deviceowner when filtering existing routerinterfaces @zetaab
ignore openstack managed volume tags @zetaab
kops version: Add —short flag, use it to get version in scripts @justinsb
find sshkey resource when updating cluster @zetaab
implement GetCloudGroups for openstack @zetaab
minor fixes to openstack @zetaab
fix openstack lb pool member logic @zetaab
Support “egress: External” to avoid configuring networking @justinsb,,@moustafab
Bump alpha channels @mikesplain
Update bazel rules @mikesplain
implement delete cluster for openstack @zetaab
Openstack Floating IP Deletion @drekle
update openstack documentation @zetaab
Updated OWNERS file to include link to docs @rlenferink
[jjo] add docker-ce 18.06.2 for CVE-2019-5736 @jjo
Add permission for CreateTag on ENI to amazon-vpc-cni-k8s @nak3
Document etcd3 migration process @justinsb
Normalize etcd cluster provider names @justinsb
Support etcd-manager v3, suitable for backporting @justinsb
Openstack loadbalancers erronous modification requests @drekle
fix typos for addon doc @fatsheep9146
upgrade calico to 2.6.12 to fix TTA-2018-001 @mechpen
Use the forward plugin instead of proxy plugin in CoreDNS @rajansandeep
Update bazel workspace @mikesplain
Fix machine types and cleanup makefile @mikesplain
Add jessie patch @jjo, #6461
Allow NodeAuthorizer to speak via HTTP Proxy if configured #6468
Updated Canal manifest to v3.5.0 for k8s v1.12+ #6469
Update document for GPU support #6246
Fixing kops-4049 #6210
kube-apiserver: Add oidc-required-claim flag #6453
add OWNERS file to openstack spesific folders #6367
Update Loadbalancer Pools #6433
fix hostnames in kops openstack #6442
implement ig deletegroup for openstack #6418
Removing openstack credential file support #6480
fix error when updating/creating lb in openstack #6431
recheck floatingip after server is active #6432
Ability to scale down instancegroup in openstack #6421
expose DryRunTarget changes and deletions #6415
support both octavia and old lbaasv2 api in openstack #6438
Guess SSH usernames for RHEL & Centos in toolbox dump #6487
Choose docker version 18.06.2 for k8s >= 1.12 #6488
Install kubelet config for default centos user #6489
Update the CoreDNS manifest #6485
docs: improve the queries for finding RHEL/CentOS images #6486
Workaround for overlay2 vs rhel-family docker bug #6491
retry l3floatingip list in fresh cluster #6497
Update 1.12 addon manifests to use apps/v1, rbac v1 #6397
Fix package name & version for container-selinux #6492
AWS Mixed Instances Policy / Fleet #6277
Adding Comment #6508
Kube Proxy Metrics Option #6513
Sprig (Toolbox Templating) #6515
Etcd memory and cpu requests #6313
Map docker 18.06.3 #6523
Make docker 18.06.3 the default for k8s >= 1.12 #6524
Document strategy for cve_2019_5736 #6522
Try using chattr to mark docker-runc as immutable #6506
Simple mirror support #6503
Bump etcd-manager version to 3.0.20190224 #6526
update gophercloud vendor dependencies #6478
specify dns servers to openstack subnet #6530
possibility to specify floatingip subnet for resources in openstack #6477
Add Experimental Cluster Signing Duration flag #6525
set net.ipv4.ip_local_reserved_ports to the KubeAPIServer ServiceNodePortRange parameter on nodeup #6343
spread instances equally to all AZs #6534
update-machine-types: more metal instance types #6551
Add changelist for 1.11.1 #6565
Fix panic when using etcd-manager and resource requests are nil #6563
Promote Kubernetes 1.11.7 to stable #6566
Upgrade alpha to latest #6568
implement delete instance, this is needed in rolling-update #6576
Stop setting deprecated —allow-privileged Kubelet flag in 1.14 #6340
Openstack Security Group hardening #6521
Update embargo doc link in SECURITY_CONTACTS and change PST to PSC #6601
Instance LaunchConfig/Template Bug Fix #6590
add docker.insecureRegistries flag #6586
Add line breaks in example release cycle #6591
[jjo] Update Weave Net to version 2.5.1 #6370
Adding installation guidelines for Windows #6594
Remove confusing comma in README #6607
Add ServiceAccountKeyFile to KubeAPIServerConfig #6578
moving chrisz100 to approver level #6434
Fix dashboard yaml that returned 404 #6479
Replace Y / N Markings of Compatibility Matrix in readme with ✔ / ❌ #6539
Rename addon.yml to addon.yaml #6323
addons/cluster-autoscaler: Add jq installation for OSX environment #6567
Update docs on authentication #6575
Omit IP-in-IP protocols in Openstack CNI Rules #6614
External out-of-tree CloudControllerManager support for openstack #6444
Use EnsureTask for create static pod directory #6616
Fix documentation about targetGroupArn key #6611
Update rolling_update.md ,@Pharb
fix typo @zqm19941101
Correcly handle CRLF in the manifest @gtrafimenkov
Fix confusing k8s upgrade docs for Terraform users @tspacek, #6275
Added Audit Webhook config ,@jpbelangerupgrade
Spotinst: Avoid spurious changes @liranp
Fix amazon-vpc-routed-eni yaml template @tvi
Replace gcr.io URL with k8s.gcr.io vanity URL @justinsb
support gossip for AliCloud @LilyFaFa
add natGateways tasks for ALICloud @LilyFaFa
Fix some of the docker package names & versions @justinsb
Apply scope fix in #6502 to all manifest versions @tvi, #6622
Add —kubeconfig flag to kops export kubecfg #5955
add support to set cluster spec.kubelet #6619
Upgrade bazel gazelle #6609
Fix typo #6621
Support g3s for gpu driver installation #6538
Fix docker-healthcheck to work around Docker bug. #6448
docs: create checklist for new kubernetes version #5818
Fix metrics server addon #6201
Always create /var/lib/kubelet, even in bootstrap mode #5982
Launch Template Feature Flag #6512
Remove docker-prestart hook #6564
kops 1.12 configuration for calico: use CRDs #6358
Quick Clean #6634
Sync data-types for webhook config with upstream #6626
Add manage security groups for loadbalancers #6632
Enable etcd-manager / etcd3 / etcd-tls in kops 1.12 #6359
Use EnsureTask for internal api route53 record #6629
Added reminder to publish conformance results in release process #6640
Update aws-china.md #6643
Openstack server name collisions #6650
tiny backslash arrangement #6652
Openstack environment escaping #6657
Update upgrade.md #6654
add ALI flags #6628
Override volume zone name #6655
Updated Flannel manifest to 0.11.0 #6660
Update flannel version in bootstrapchannelbuilder ,@justinsb
Add flags for TLS Cipher suites customization for API Server, Kubelet and Controller-Manager @rochacon
If using etcd-backup and TLS is enabled, pass relevant options @KashifSaadat
Bump etcd-manager / etcd-backup to 3.0.20190325 @justinsb
2048 - Add cloudLabels as tags to API ELB resource @ryan-dyer
Bump K8s 1.11 to 1.11.9 in the alpha channel @olemarkus
Upgrade rules go @mikesplain
Fix a missing dep lock @mikesplain

1.12.0-alpha.1 to 1.12.0-alpha.2

Support download protokube from mirror #6673
Promote alpha to stable and update alpha #6669
Upload protokube to github as part of release #6674
Use CNI 0.7.5 #6671
Put 1.12 into stable channel, for users of kops 1.12-alphas #6672
Support mirrors with restricted characters #6675

1.12.0-alpha.2 to 1.12.0-alpha.3

Fix Key error change Overrides to Override @granular-ryanbonham
Add selector back to calico 1.12 deployment @justinsb
Update etcd-manager to 1.0.20190328 @justinsb

1.12.0-alpha.3 to 1.12.0-beta.1

Fix tagging and remove tagging elbs #6705
Add DNS Resource Settings #6731
Update instances types #6733
Update kube-dns 1.3.0 to 1.3.3 #6734
kube-dns-autoscaler: Add node watch to permissions #6740
Increase apiserver timeout to 45 seconds #6743
Fix issue #6700: User Data for launch templates & other terraform issues #6732

1.12.0-beta.1 to 1.12.0-beta.2

kube-dns: Update to 1.14.13 @justinsb
Launch Template use version number as well as name.@granular-ryanbonham
use dynamic s3 prefix in addAmazonVPCCNIPermissions func @bksteiny

1.12.0-beta.2 to 1.12.0

IAM Permission to Support Scaling from 0 with Lauch Templates #6861
Avoid concurrent write corruption to /etc/hosts ,@granular-ryanbonham
Add i3en instance types @mikesplain
Add t3a family @mikesplain
Use existing SSHKeyName if no public key is created. @rralcala
bazel: fix distroless imports for latest bazel @justinsb
pkg/model: Fix dropped error @alrs
Add ability to specify cpuRequest for API Server @granular-ryanbonham
KubeAPIServer HTTP2 Stream Parameter @gambol99
Add support for AWS ap-east-1 region @wxdao
Add min-resync-period for Controller Manager @maruina
Allow the AWS IAM Authenticator image name to be overridden @rifelpet
Add cpu management policy config @lynchc
Carry Provisioned IOPS to Terraform and CloudFormation templates @MathieuMailhos
update tolerations to openstack external cloud provider @zetaab
Fix typo in aws-iam-authenticator image field name @rifelpet
add the registry-qps kubelet flag @sp-joseluis-ledesma
Deep-copy proto state to prevent concurrent modification @justinsb
Publish utils.tar.gz to github releases also @justinsb
Allow uneven etcd zones @adammw
Add terraform support for additional CIDR blocks. @rdrgmnzs
Canal manifest updates for k8s v1.12+ @KashifSaadat
Update to etcd-manager 1.0.20190509 @justinsb
S3 VFS: Default to current region from metadata service @justinsb, #6943
etcd-manager: Update to 3.0.20190513 #6959
Fix Docker not being installed on Ubuntu 16.04 #6965
Issue #6945 ,@mikesplain

1.12.0 to 1.12.1

Don’t panic when deleting instancegroups #7000
etcd-manager: update to 3.0.20190516 #7007
Terraform: fix options field, should be spot_options #6988

1.12.1 to 1.12.2

Mark ENI 0 as delete_on_termination for LaunchTemplates @granular-ryanbonham

1.12.2 to 1.12.3

Cherry pick of #7211: Use NodeAuthorizer config options instead of soely #7232
Cherry pick of #7219: Make an actual deep-copy of the state #7235
Upgrade Calico to 3.7.2 #7051
Update canal to 3.6.4, for TTA-2019-002 #7275
Bumping calico to 3.7.4. #7249
Cherry pick of #7185: Replace behavior for aws hostnameOverride #7308
Calico -> 3.7.4 for older versions #7282
Warn/prevent if the version of etcd is unsupported with etcd-manager #7340