RFC: Coordinated dynamic storage version #1201

caesarxuchao · 2019-08-06T00:38:33Z

Superseded by #1220

An alternative (hopefully a better one) to #1176.

We propose a mechanism for HA API servers to vote on the storage version to use, then show the selected storage version in the discovery document and switch to use the selected version atomically.

/assign @yliaog @wojtek-t @lavalamp
Thank you for you previous comments in #1176

/sig api-machinery

wojtek-t · 2019-08-06T07:19:27Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+
+### Curating a list of participating API servers in HA master
+
+Currently, API servers maintain such a list in the "kubernetes" endpoints. See


We have a way to do that - this doesn't mean we always do.
For example in GKE, this endpoint contains only IP of the LB in front of apiservers (whether that's good or bad is a separate discussion, but I guess we may not be the only one doing something like that).

I agree that other operators might have other forms of deployments that also hide the individual apiserver endpoints, so we can't rely on the current "kubernetes" service endpoints to get a reliable list of apiservers.

As I mentioned in the KEP, I will send another KEP to formalize the procedure of apiserver census. Operators will need to configure an ID for each apiserver to make it work.

wojtek-t · 2019-08-06T07:20:36Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+    participating API server. This means the API server has gone and the entry
+    is stale. Removes such entries.
+  * checks if all participating API server votes for the same version. If so,
+    sets the version as the Selectedversion.


What if not all of them are voting for the same version?

The preferred version in the first vote becomes the first selected version.

Afterwards, only after all apiservers prefer a new version, then it becomes the new selected version.

wojtek-t · 2019-08-06T07:24:55Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+checks if key does not already exist before committing a create operation. As
+another example, kube-apiserver checks that the resourceVersion is the latest
+before committing an update operation. Adding another transaction condition does
+not add extra RPC call.


It would be good to run some benchmark of it to understand the impact of it. I hope it should be relatively small, but I don't know etcd internals well enough to know exactly what would happen.
@jpbetz - FYI

wojtek-t · 2019-08-06T07:26:55Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+version. The storage version in the CRD spec is the "selected storage version".
+
+The current implementation of the apiextension-apiserver already provides the
+"loose guarantee".


Do we have any signals from users that it's not enough?
If not, maybe it's not worth changing that (and maybe that also means it should be good enough for builtin resources).

No, I haven't heard any complain.

I'm willing to pursue the "loose guarantee" path if all reviewers think it's good enough.

yliaog · 2019-08-06T20:32:57Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+* gets the StorageVersion object for this resource,
+  * if the object does not exist, creates one, setting this kube-apiserver's
+  preferred storage version as the Selectedversion, also adding its preferred
+  storage version to the CandidateVersions list. Then jumps to the last step.


last step is which step? probably better to number the steps

yliaog · 2019-08-06T20:38:50Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+* gets the list of all participating API servers,
+* gets all the StorageVersion objects,
+* removes the StorageVersion object if none of its CandidateVersions.APIServerID
+  is a participating API server.


usually api servers are independent, unaware of the existence of the other api servers. what about using keep-alive have each api server to tell it is still alive, participating

Not sure what you meant by "keep-alive", is it a formal kubernetes API?

The way Kubernetes curating the "kubernetes" endpoints today is using a TTL mechanism (design, code), is it what you are suggesting?

Yes, that is it.

yliaog · 2019-08-06T21:03:29Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+   exists in all parts of Kubernetes API. Client-side coordination is necessary
+   to overcome the race. Let's take the storage migrator as an example. For the
+   storage migrator to work properly, the required client-side coordination is
+   that at most one master version change could happen in any 5 minute window.


i don't think this is client-side coordination, this is an assumption / guarantee made on the server side, i.e., cluster admins do not do version upgrade or downgrade more than once every 5 minutes.

k8s-ci-robot · 2019-08-09T20:41:06Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: caesarxuchao
To complete the pull request process, please assign lavalamp
You can assign the PR to them by writing /assign @lavalamp in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-api-machinery/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

caesarxuchao · 2019-08-09T21:30:10Z

/assign @deads2k

liggitt · 2019-08-13T14:24:23Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+server will change the selected version to the new storage version, and all API
+servers start to use the new version.
+
+### The selected storage version is supported by all API servers during upgrades/downgrades


This describes selecting which version to use for a type which all API servers support, but at different versions.

Would the same mechanism correctly handle a type which some API servers do not support at all (e.g. avoid serving a new beta type while there are joined API servers that do not make that type available, xref kubernetes/kubernetes#81286)

I think so. It's mentioned in the "future work" section: "For example,
kube-apiservers can vote on what API is served, and only serve the API that is
supported by all servers."

deads2k · 2019-08-13T14:46:04Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+
+During the rolling update of an HA master, API server instances *i)* use
+different storage versions for a built-in API resource, and *ii)* show different
+storageVersionHash in the discovery documents. These facts make the storage


elaborate here. I suspect it's because etcd may contain records written in different storage versions.

deads2k · 2019-08-13T14:47:13Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+[storageVersionHash]:https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L979
+[details]:https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/35-storage-version-hash.md#ha-masters
+
+We propose a mechanism for HA API servers to vote on the storage version to use,


I don't think that we need to have all apiserver allow configurable storage. Instead, an apiserver can indicate which storage version it writes (hardcoded like today) and the entity driving the storage migrator can decide to wait until all apiservers agree. That builds one system on another instead of changing the underlying system.

deads2k · 2019-08-13T14:48:24Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+
+```golang
+// Storage version for a specific resource.
+type StorageVersion struct {


I would like to see a spec/status split here. apiservers write the status of what is

groupversions this server can read

groupversions this server will write

I don't see any fields for spec yet.

Is "groupversions this server can read" essential to this API? That information is beyond "storageVersion". I'll need to come up with a new API name.

deads2k · 2019-08-13T14:48:43Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+### Storage Version API
+
+We introduce a new API `StorageVersion`, in a new API group
+`kube-apiserver.internal.k8s.io/v1alpha1`. 


this isn't specific to kube-apiserver. This is generic for any aggregated apiserver as well.

deads2k · 2019-08-13T14:49:39Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+
+type VersionCandidate struct {
+  // The ID of the reporting API server. 
+  APIServerID string


how will old/outdated servers be stripped removed? A server can crash and be replaced by a different server with a different ID. Imagine using an autoscaling deployment as a for instance.

deads2k · 2019-08-13T14:54:22Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+this [design doc][] for details.
+
+We will formalize the API in another KEP. For the purpose of this KEP, all we
+need to know is that HA master is able to maintain a list of participating API


ah, neat. This is how you handle pruning. Combined with visibility into apiservices, this is sufficient to selectively prune apiservers.

deads2k · 2019-08-13T14:56:20Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+
+[design doc]:https://github.com/kubernetes/community/pull/939
+
+### Voting for Storage Version


I'm not seeing why voting is necessary for the storage migrator to function correctly. It seems like being able to see the current state and knowing whether it is consistent across the cluster is sufficient to make a choice about migrating that cluster. That, in combination with a resource version seems fully sufficient to know if storage migration should commence and whether it was successful.

sanchezl

Overall I think this proposal should be about discovery, not configuration, enabling other processes to drive the storage migration.

sanchezl · 2019-08-13T15:27:19Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+[storageVersionHash]:https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L979
+[details]:https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/35-storage-version-hash.md#ha-masters
+
+We propose a mechanism for HA API servers to vote on the storage version to use,


The StorageVersionHash problem linked above is about being able to detect when its safe to trigger the storage migratior, whereas here you are suggesting you intent to actually configure which storage version to use. Please clarify the intended use. I would suggest this proposal should only be about discovering which storage versions are in use.

sanchezl · 2019-08-13T15:29:22Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+  ObjectMeta
+  // The selected storage version. All API server intances encode the resource
+  // objects in this version when committing it to etcd.
+  SelectedVersion string


If we are not configuring storage (I don't think this is what we want), this field is unnecessary.

sanchezl · 2019-08-13T15:33:42Z

keps/sig-api-machinery/20190802-dynamic-coordinated-storage-version.md

+  APIServerID string
+  // The preferred storage version of the reporting API server. The preferred
+  // storage version is hardcoded in each Kubernetes release.
+  PreferredVersion string


This should simply be the storage version in use by this api server.

HA storage version

0c034e9

k8s-ci-robot assigned lavalamp, wojtek-t and yliaog Aug 6, 2019

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 6, 2019

caesarxuchao mentioned this pull request Aug 6, 2019

RFC: adding a KEP for consistent discovery documents in HA setup #1176

Closed

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 6, 2019

k8s-ci-robot requested review from deads2k and lavalamp August 6, 2019 00:38

k8s-ci-robot added the kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory label Aug 6, 2019

wojtek-t reviewed Aug 6, 2019

View reviewed changes

yliaog reviewed Aug 6, 2019

View reviewed changes

caesarxuchao force-pushed the HA-storage-version branch from b6cd34b to ff8ad27 Compare August 9, 2019 20:42

updates

6700cbb

caesarxuchao force-pushed the HA-storage-version branch from ff8ad27 to 6700cbb Compare August 9, 2019 21:26

k8s-ci-robot assigned deads2k Aug 9, 2019

liggitt mentioned this pull request Aug 13, 2019

1.14 masters flapping until all are upgraded kubernetes/kubernetes#81286

Closed

liggitt reviewed Aug 13, 2019

View reviewed changes

deads2k reviewed Aug 13, 2019

View reviewed changes

sanchezl suggested changes Aug 13, 2019

View reviewed changes

caesarxuchao mentioned this pull request Aug 22, 2019

StorageVersion API for HA master #1220

Merged

caesarxuchao closed this Aug 23, 2019

sanchezl mentioned this pull request Mar 19, 2020

REQUEST: New membership for sanchezl kubernetes/org#1730

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Coordinated dynamic storage version #1201

RFC: Coordinated dynamic storage version #1201

caesarxuchao commented Aug 6, 2019 •

edited

Loading

wojtek-t Aug 6, 2019

caesarxuchao Aug 6, 2019 •

edited

Loading

wojtek-t Aug 6, 2019

caesarxuchao Aug 6, 2019

wojtek-t Aug 6, 2019

wojtek-t Aug 6, 2019

caesarxuchao Aug 6, 2019

yliaog Aug 6, 2019

yliaog Aug 6, 2019

caesarxuchao Aug 6, 2019

yliaog Aug 7, 2019

yliaog Aug 6, 2019

k8s-ci-robot commented Aug 9, 2019

caesarxuchao commented Aug 9, 2019

liggitt Aug 13, 2019

caesarxuchao Aug 14, 2019

deads2k Aug 13, 2019

deads2k Aug 13, 2019

deads2k Aug 13, 2019

caesarxuchao Aug 20, 2019

deads2k Aug 13, 2019

deads2k Aug 13, 2019

deads2k Aug 13, 2019

deads2k Aug 13, 2019

sanchezl left a comment

sanchezl Aug 13, 2019

sanchezl Aug 13, 2019

sanchezl Aug 13, 2019


		### Curating a list of participating API servers in HA master

		Currently, API servers maintain such a list in the "kubernetes" endpoints. See


		[design doc]:https://github.com/kubernetes/community/pull/939

		### Voting for Storage Version

RFC: Coordinated dynamic storage version #1201

RFC: Coordinated dynamic storage version #1201

Conversation

caesarxuchao commented Aug 6, 2019 • edited Loading

Choose a reason for hiding this comment

caesarxuchao Aug 6, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Aug 9, 2019

caesarxuchao commented Aug 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanchezl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caesarxuchao commented Aug 6, 2019 •

edited

Loading

caesarxuchao Aug 6, 2019 •

edited

Loading