Skip to content

Commit

Permalink
Merge pull request #20369 from jimangel/fix-version-skew
Browse files Browse the repository at this point in the history
Automating Version Skew
  • Loading branch information
k8s-ci-robot committed Apr 17, 2020
2 parents 3675821 + a3decc2 commit 8e2544f
Show file tree
Hide file tree
Showing 3 changed files with 40 additions and 33 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -609,13 +609,10 @@ of Pod network add-ons.

## Version skew policy {#version-skew-policy}

The `kubeadm` tool of version vX.Y may deploy clusters with a control plane of version vX.Y or vX.(Y-1).
`kubeadm` vX.Y can also upgrade an existing kubeadm-created cluster of version vX.(Y-1).
The `kubeadm` tool of version v{{< skew latestVersion >}} may deploy clusters with a control plane of version v{{< skew latestVersion >}} or v{{< skew prevMinorVersion >}}.
`kubeadm` v{{< skew latestVersion >}} can also upgrade an existing kubeadm-created cluster of version v{{< skew prevMinorVersion >}}.

Due to that we can't see into the future, kubeadm CLI vX.Y may or may not be able to deploy vX.(Y+1) clusters.

Example: `kubeadm` v1.8 can deploy both v1.7 and v1.8 clusters and upgrade v1.7 kubeadm-created clusters to
v1.8.
Due to that we can't see into the future, kubeadm CLI v{{< skew latestVersion >}} may or may not be able to deploy v{{< skew nextMinorVersion >}} clusters.

These resources provide more information on supported version skew between kubelets and the control plane, and other Kubernetes components:

Expand Down
54 changes: 27 additions & 27 deletions content/en/docs/setup/release/version-skew-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Kubernetes versions are expressed as **x.y.z**,
where **x** is the major version, **y** is the minor version, and **z** is the patch version, following [Semantic Versioning](http://semver.org/) terminology.
For more information, see [Kubernetes Release Versioning](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#kubernetes-release-versioning).

The Kubernetes project maintains release branches for the most recent three minor releases.
The Kubernetes project maintains release branches for the most recent three minor releases ({{< skew latestVersion >}}, {{< skew prevMinorVersion >}}, {{< skew oldestMinorVersion >}}).

Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility.
Patch releases are cut from those branches at a regular cadence, or as needed.
Expand All @@ -41,82 +41,82 @@ In [highly-available (HA) clusters](/docs/setup/production-environment/tools/kub

Example:

* newest `kube-apiserver` is at **1.13**
* other `kube-apiserver` instances are supported at **1.13** and **1.12**
* newest `kube-apiserver` is at **{{< skew latestVersion >}}**
* other `kube-apiserver` instances are supported at **{{< skew latestVersion >}}** and **{{< skew prevMinorVersion >}}**

### kubelet

`kubelet` must not be newer than `kube-apiserver`, and may be up to two minor versions older.

Example:

* `kube-apiserver` is at **1.13**
* `kubelet` is supported at **1.13**, **1.12**, and **1.11**
* `kube-apiserver` is at **{{< skew latestVersion >}}**
* `kubelet` is supported at **{{< skew latestVersion >}}**, **{{< skew prevMinorVersion >}}**, and **{{< skew oldestMinorVersion >}}**

{{< note >}}
If version skew exists between `kube-apiserver` instances in an HA cluster, this narrows the allowed `kubelet` versions.
{{</ note >}}

Example:

* `kube-apiserver` instances are at **1.13** and **1.12**
* `kubelet` is supported at **1.12**, and **1.11** (**1.13** is not supported because that would be newer than the `kube-apiserver` instance at version **1.12**)
* `kube-apiserver` instances are at **{{< skew latestVersion >}}** and **{{< skew prevMinorVersion >}}**
* `kubelet` is supported at **{{< skew prevMinorVersion >}}**, and **{{< skew oldestMinorVersion >}}** (**{{< skew latestVersion >}}** is not supported because that would be newer than the `kube-apiserver` instance at version **{{< skew prevMinorVersion >}}**)

### kube-controller-manager, kube-scheduler, and cloud-controller-manager

`kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` must not be newer than the `kube-apiserver` instances they communicate with. They are expected to match the `kube-apiserver` minor version, but may be up to one minor version older (to allow live upgrades).

Example:

* `kube-apiserver` is at **1.13**
* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are supported at **1.13** and **1.12**
* `kube-apiserver` is at **{{< skew latestVersion >}}**
* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are supported at **{{< skew latestVersion >}}** and **{{< skew prevMinorVersion >}}**

{{< note >}}
If version skew exists between `kube-apiserver` instances in an HA cluster, and these components can communicate with any `kube-apiserver` instance in the cluster (for example, via a load balancer), this narrows the allowed versions of these components.
{{< /note >}}

Example:

* `kube-apiserver` instances are at **1.13** and **1.12**
* `kube-apiserver` instances are at **{{< skew latestVersion >}}** and **{{< skew prevMinorVersion >}}**
* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` communicate with a load balancer that can route to any `kube-apiserver` instance
* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are supported at **1.12** (**1.13** is not supported because that would be newer than the `kube-apiserver` instance at version **1.12**)
* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are supported at **{{< skew prevMinorVersion >}}** (**{{< skew latestVersion >}}** is not supported because that would be newer than the `kube-apiserver` instance at version **{{< skew prevMinorVersion >}}**)

### kubectl

`kubectl` is supported within one minor version (older or newer) of `kube-apiserver`.

Example:

* `kube-apiserver` is at **1.13**
* `kubectl` is supported at **1.14**, **1.13**, and **1.12**
* `kube-apiserver` is at **{{< skew latestVersion >}}**
* `kubectl` is supported at **{{< skew nextMinorVersion >}}**, **{{< skew latestVersion >}}**, and **{{< skew prevMinorVersion >}}**

{{< note >}}
If version skew exists between `kube-apiserver` instances in an HA cluster, this narrows the supported `kubectl` versions.
{{< /note >}}

Example:

* `kube-apiserver` instances are at **1.13** and **1.12**
* `kubectl` is supported at **1.13** and **1.12** (other versions would be more than one minor version skewed from one of the `kube-apiserver` components)
* `kube-apiserver` instances are at **{{< skew latestVersion >}}** and **{{< skew prevMinorVersion >}}**
* `kubectl` is supported at **{{< skew latestVersion >}}** and **{{< skew prevMinorVersion >}}** (other versions would be more than one minor version skewed from one of the `kube-apiserver` components)

## Supported component upgrade order

The supported version skew between components has implications on the order in which components must be upgraded.
This section describes the order in which components must be upgraded to transition an existing cluster from version **1.n** to version **1.(n+1)**.
This section describes the order in which components must be upgraded to transition an existing cluster from version **{{< skew prevMinorVersion >}}** to version **{{< skew latestVersion >}}**.

### kube-apiserver

Pre-requisites:

* In a single-instance cluster, the existing `kube-apiserver` instance is **1.n**
* In an HA cluster, all `kube-apiserver` instances are at **1.n** or **1.(n+1)** (this ensures maximum skew of 1 minor version between the oldest and newest `kube-apiserver` instance)
* The `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` instances that communicate with this server are at version **1.n** (this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version)
* `kubelet` instances on all nodes are at version **1.n** or **1.(n-1)** (this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version)
* In a single-instance cluster, the existing `kube-apiserver` instance is **{{< skew prevMinorVersion >}}**
* In an HA cluster, all `kube-apiserver` instances are at **{{< skew prevMinorVersion >}}** or **{{< skew latestVersion >}}** (this ensures maximum skew of 1 minor version between the oldest and newest `kube-apiserver` instance)
* The `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` instances that communicate with this server are at version **{{< skew prevMinorVersion >}}** (this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version)
* `kubelet` instances on all nodes are at version **{{< skew prevMinorVersion >}}** or **{{< skew oldestMinorVersion >}}** (this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version)
* Registered admission webhooks are able to handle the data the new `kube-apiserver` instance will send them:
* `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` objects are updated to include any new versions of REST resources added in **1.(n+1)** (or use the [`matchPolicy: Equivalent` option](/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchpolicy) available in v1.15+)
* The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in **1.(n+1)**
* `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` objects are updated to include any new versions of REST resources added in **{{< skew latestVersion >}}** (or use the [`matchPolicy: Equivalent` option](/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchpolicy) available in v1.15+)
* The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in **{{< skew latestVersion >}}**

Upgrade `kube-apiserver` to **1.(n+1)**
Upgrade `kube-apiserver` to **{{< skew latestVersion >}}**

{{< note >}}
Project policies for [API deprecation](/docs/reference/using-api/deprecation-policy/) and
Expand All @@ -128,17 +128,17 @@ require `kube-apiserver` to not skip minor versions when upgrading, even in sing

Pre-requisites:

* The `kube-apiserver` instances these components communicate with are at **1.(n+1)** (in HA clusters in which these control plane components can communicate with any `kube-apiserver` instance in the cluster, all `kube-apiserver` instances must be upgraded before upgrading these components)
* The `kube-apiserver` instances these components communicate with are at **{{< skew latestVersion >}}** (in HA clusters in which these control plane components can communicate with any `kube-apiserver` instance in the cluster, all `kube-apiserver` instances must be upgraded before upgrading these components)

Upgrade `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` to **1.(n+1)**
Upgrade `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` to **{{< skew latestVersion >}}**

### kubelet

Pre-requisites:

* The `kube-apiserver` instances the `kubelet` communicates with are at **1.(n+1)**
* The `kube-apiserver` instances the `kubelet` communicates with are at **{{< skew latestVersion >}}**

Optionally upgrade `kubelet` instances to **1.(n+1)** (or they can be left at **1.n** or **1.(n-1)**)
Optionally upgrade `kubelet` instances to **{{< skew latestVersion >}}** (or they can be left at **{{< skew prevMinorVersion >}}** or **{{< skew oldestMinorVersion >}}**)

{{< warning >}}
Running a cluster with `kubelet` instances that are persistently two minor versions behind `kube-apiserver` is not recommended:
Expand Down
10 changes: 10 additions & 0 deletions layouts/shortcodes/skew.html
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
<!-- subtract latest 1 minor number for "prevMinorVerison" -->
{{- $prevMinorVersion := sub $minorVersion 1 -}}

<!-- subtract latest 2 minor numbers for "oldestMinorVerison" -->
{{- $oldestMinorVersion := sub $minorVersion 2 -}}

<!-- output nextMinorVersion based on captured arg -->
{{- if eq $version "nextMinorVersion" -}}
{{- $nextMinorVersion := printf "%s.%d" (index $versionArray 0) $nextMinorVersion -}}
Expand All @@ -34,9 +37,16 @@
{{- $prevMinorVersion -}}
{{- end -}}

<!-- output oldestMinorVersion based on captured arg -->
{{- if eq $version "oldestMinorVersion" -}}
{{- $oldestMinorVersion := printf "%s.%d" (index $versionArray 0) $oldestMinorVersion -}}
{{- $oldestMinorVersion -}}
{{- end -}}

<!--
example shortcode use:
- skew nextMinorVersion
- skew latestVersion
- skew prevMinorVersion
- skew oldestMinorVersion
-->

0 comments on commit 8e2544f

Please sign in to comment.