CRD versioning design proposal #2026

mbohlool · 2018-04-10T22:43:22Z

CRD versioning proposal is discussed in detail here:

https://docs.google.com/document/d/1Ucf7JwyHpy7QlgHIN2Rst_q6yT0eeN9euzUV6kte6aY/edit

This is the main part of the design. Look at #2054 first.

k8s-ci-robot · 2018-04-10T22:43:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: deads2k

Assign the PR to them by writing /assign @deads2k in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

contributors/design-proposals/api-machinery/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sttts · 2018-04-11T10:16:49Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+# **Background**
+
+Custom Resource Definitions ([CRDs](https://kubernetes.io/docs/concepts/api-extension/custom-resources/)) are a popular mechanism for extending Kubernetes, due to their ease of use compared with the main alternative of building an Aggregated API Servers. They are, however, lacking a very important feature that all other kubernetes objects support: Versioning. Today, each CR can only have one version and there is no clear way for authors to advance their resources to a newer version other than creating a completely new CRD and converting everything manually in sync with their client software.


nit: server

sttts · 2018-04-11T10:17:53Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+* Support ability to change storage version
+
+* Support Validation/OpenAPI schema for all versions


for each or for all at once?

Added a clarification.

sttts · 2018-04-11T10:18:55Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+The summary is to support a list of versions that will include current version. One of these versions can be flagged as the storage version and all versions ever marked as storage version will be listed in a stored_version field in the Status object to enable authors to plan a migration for their stored objects.
+
+The current `Version` field in planned to be deprecated in a later release and will be used to pre-populate the `Versions` field (The `Versions` field will be defaulted to a single version, constructed from top level `Version` field). The `Version` field will be also be mutable to give a way to the authors to remove it from the list.


sttts · 2018-04-11T10:23:11Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+  // ***************
+  // Versions is the list of all supported versions for this resource.
+  // Validation: All versions must use the same validation schema for now. i.e., top 
+  // level Validation field is applied to all of these versions.


which means you can use AnyOf: [{ properties: { APIVersion: { pattern: "group/v2" }}}, ...] to have different validations for each version.

note that people will use this way and we have no way to deprecate the top-level validation ever. We could instead add validation to the versions and enforce its use as soon as there is more than one version defined.

The same is true for subresources. Should we move them to Versions as well? I think we should. Today there is no graceful way to enable the /status subresource without breaking either old or new clients.

sttts · 2018-04-11T10:25:36Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+  Name string
+  // Served is a flag enabling/disabling this version from being served via REST APIs
+  Served Boolean
+  // Storage flags the release as a storage version. There can be only one version


There must be exactly one version flagged as Storage.

sttts · 2018-04-11T10:25:55Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+}
+```
+
+The Status object will have a list of potential stored objects. This data is necessary to do a storage migration in future (the author can choose to do the migration themselves but there is a plan [TODO(lavalamp): link?] to solve the problem of migration, potentially for both standard and custom types).


stored versions.

sttts · 2018-04-11T10:28:45Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+Basic validations needed for the `version` field are:
+
+* `Spec.Version` field exists in `Spec.Versions` field (serve should be True for that version if there is any).


so we force at least one served version. Do we want that? Do we even have to (to allow e.g. the GC controller to do its work)?

I think we don't want that. We discussed this on the implementation PR. I will amend this.

sttts · 2018-04-11T10:30:25Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+Basic validations needed for the `version` field are:
+
+* `Spec.Version` field exists in `Spec.Versions` field (serve should be True for that version if there is any).
+* If there is any `Served` `Version` in list of `Versions`, then the `Spec.Version`’s version in the `Spec.Versions` list should have a set `Served` flag. This is for backward compatibility. An old controller expect that version to be served but only the whole CRD is served. CRD Registration controller should unregister a CRD with no serving version.


Isn't this "if" implicitly true by the "serve should be True" above?
Can spec.version be empty?

I changed previous one. So the if is not true all the time. You can disable all versions and in that case the version defined in Version field does not need to have Served flag enabled. But if we serve ANY version, then the version defined in Version field should point to a Served version.

I updated the wording to make this more clear.

sttts · 2018-04-11T10:31:33Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+  // flagged as Storage.
+  Storage Boolean
+  //
+  Conversion CustomResourceConversion


pointer with omitempty.

sttts · 2018-04-11T10:32:33Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+}
+
+type CustomResourceConversion struct {
+  Declarative map[string]CustomResourceDeclarativeConversion


what's the meaning of the string?
Are declarative conversions sorted?

They are not sorted. The map key is the target version.

sttts · 2018-04-11T10:32:59Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+type CustomResourceConversion struct {
+  Declarative map[string]CustomResourceDeclarativeConversion
+  External CustomResourceConversionWebhook


pointer and omitempty

Having a webhook definition per version looks odd. I would expect it globally with a set of support conversions pairs. Then you don't have to maintain n webhook structs. Alternatively, we could think about a global named conversion webhook field and reference the names here (like volumes and mounts in pods).

I explained the design further down. It is cleaner to define the webhook on the version but the validation will not let user define more than one webhook. It will just make sure if more than one webhook is defined, they are the same.

Don't like this kind of redundancy. Why is it cleaner?

sttts · 2018-04-11T10:33:46Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+  Conversion CustomResourceConversion
+}
+
+type CustomResourceConversion struct {


are multiple conversions mutual exclusive for each version pair?

sttts · 2018-04-11T10:39:00Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+The API structure above allows some invalid states in order to keep a clean easy API. There need to be validation to catch undesire states. The validations would be
+
+* Multiple webhooks are not allowed for a CRD. This will help us improve the performance by keeping the connection open to the webhook if the user defined a chain of version conversions.
+* There should be no loop in the conversion graph. This includes v1->v2->v1 or v1->v2->v3->v1. The former is implicit in the assumption that webhook conversion is round-trippable. The second one allows two way to convert v1 to v3 which is not desired.


Isn't a loop fine and necessary? Every path without a loop should be unique.

sttts · 2018-04-11T10:41:25Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+type RenameConversion {
+  From JsonPath
+  To JsonPath
+}


To avoid undesired renames, better use a base json path and a field rename:

type RenameConversion struct { Object JsonPath FieldName string NewFieldName string }

I was thinking we would let structural changes later (carefully) and we can prevent them with validation for now. This way we do not need to change API or add another conversion for structural changes but I am also OK if we are thinking about a MoveConversion later though I liked the end product to only have MoveConversion and not Move and Rename both.

If we call it RenameConversion, we should be strict. If we call it MoveConversion we could restrict it to renames for the moment. Both is fine with me.

Being more explicit is better I guess. I went with your RenameConversion suggestion.

sttts · 2018-04-11T10:41:52Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+* Multiple webhooks are not allowed for a CRD. This will help us improve the performance by keeping the connection open to the webhook if the user defined a chain of version conversions.
+* There should be no loop in the conversion graph. This includes v1->v2->v1 or v1->v2->v3->v1. The former is implicit in the assumption that webhook conversion is round-trippable. The second one allows two way to convert v1 to v3 which is not desired.
+* Rename operation can be limited at first to not change the structure despite the fact that `To` field is also a json path.


See suggestion above: we only want to rename fields in objects, no moves, no structure changes.

mbohlool · 2018-04-12T00:01:48Z

@sttts I added the webhook to the original document and asked people to look at it and I thought there is no objection on the design. I see most of your comments is on webhooks. We can discuss that here, but as we do not want to implement webhooks in 1.11, Do you think it is better to remove it from here and continue discussion on the Google Doc one. It is easier to discuss things there.

sttts · 2018-04-12T10:55:15Z

Do you think it is better to remove it from here and continue discussion on the Google Doc one

We can sketch webhook here, but clearly call it a sketch, i.e. to be clear that this must be worked out.

liggitt · 2018-04-12T11:11:27Z

Do you think it is better to remove it from here and continue discussion on the Google Doc one. It is easier to discuss things there.

It seems better to describe future webhook plans generally, not as API fields yet, if they are still being designed. The previous review concentrated on the parts under consideration for implementation in 1.10.

sttts · 2018-04-16T08:28:42Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+
+type CustomResourceDeclarativeConversion {
+  // This type will have collection of conversion operations. Only one of them should be set.
+  Rename RenameConversion


This have to be an array.

Why should only one of them be set?

We can have renames, moves, deletions all at the same time. We need a logic to detect conflicts during validation of course.

This can be an array or the CustomResourceDeclarativeConversion can be an array in the map. which one is better? I personally think it is more explicit to make the CustomResourceDeclarativeConversion an array then the conversion would be look like:

v1: - Rename: ... ... - Rename: ... ... - Move: ... ... ...

Also deletion cannot happened according to our API guidelines. User can rename the field to a in-name-marked deprecated to deleted field (e.g. deprecated_... or deleted_...).

deads2k · 2018-04-18T18:35:48Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+}
+
+type CustomResourceConversion struct {
+  Declarative map[string]CustomResourceDeclarativeConversion


maps aren't allowed in our API

deads2k · 2018-04-18T18:36:43Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+  Conversion CustomResourceConversion
+}
+
+type CustomResourceConversion struct {


This is more involved than the "no changes" we had talked about before. How about we start small?

The first PR will be no change, but for 1.11 we want at least rename. Do you have any objection to the rename design here?

deads2k · 2018-04-18T18:38:28Z

contributors/design-proposals/api-machinery/customresources-versioning.md

+* None of the `Status.StoredVersion` can be removed from `Spec.Versions` list.
+* Only one of the versions in `spec.Versions` can flag as `Storage` version.
+
+# **Declarative conversion**


Ah, ok. I think I see what happened. You're saying "these are some thoughts I had", not "I want to build this an merge it in 1.11". If that's the case, can you add a "Declarative conversion design not approved in this document" in big bold letters leading the section?

Actually these are the things I want to have in 1.11 (only rename). Here we can discuss it if you have any thoughts. These are designed I discussed in one form or another with Stephan, Daniel, and Eric but none approved it yet. I guess that is the goal of this PR.

k8s-ci-robot · 2018-04-26T22:53:42Z

@mbohlool: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 10, 2018

k8s-ci-robot added kind/design Categorizes issue or PR as related to design. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Apr 10, 2018

k8s-ci-robot requested review from deads2k and lavalamp April 10, 2018 22:43

mbohlool force-pushed the crd-versioning branch from 6414803 to 9319315 Compare April 10, 2018 22:46

CRD versioning design proposal

b30582b

mbohlool force-pushed the crd-versioning branch from 9319315 to b30582b Compare April 10, 2018 22:52

sttts reviewed Apr 11, 2018

View reviewed changes

mbohlool mentioned this pull request Apr 13, 2018

Custom Resource Definition Versioning kubernetes/enhancements#544

Closed

sttts reviewed Apr 16, 2018

View reviewed changes

nikhita mentioned this pull request Apr 17, 2018

CustomResourceDefinitions kubernetes/enhancements#95

Closed

deads2k reviewed Apr 18, 2018

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 26, 2018

mbohlool closed this May 21, 2018


		# Background

		Custom Resource Definitions ([CRDs](https://kubernetes.io/docs/concepts/api-extension/custom-resources/)) are a popular mechanism for extending Kubernetes, due to their ease of use compared with the main alternative of building an Aggregated API Servers. They are, however, lacking a very important feature that all other kubernetes objects support: Versioning. Today, each CR can only have one version and there is no clear way for authors to advance their resources to a newer version other than creating a completely new CRD and converting everything manually in sync with their client software.


		* Support ability to change storage version

		* Support Validation/OpenAPI schema for all versions


		The summary is to support a list of versions that will include current version. One of these versions can be flagged as the storage version and all versions ever marked as storage version will be listed in a stored_version field in the Status object to enable authors to plan a migration for their stored objects.

		The current `Version` field in planned to be deprecated in a later release and will be used to pre-populate the `Versions` field (The `Versions` field will be defaulted to a single version, constructed from top level `Version` field). The `Version` field will be also be mutable to give a way to the authors to remove it from the list.


		Basic validations needed for the `version` field are:

		* `Spec.Version` field exists in `Spec.Versions` field (serve should be True for that version if there is any).

CRD versioning design proposal #2026

CRD versioning design proposal #2026

Conversation

mbohlool commented Apr 10, 2018 • edited Loading

k8s-ci-robot commented Apr 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbohlool commented Apr 12, 2018

sttts commented Apr 12, 2018

liggitt commented Apr 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 26, 2018

mbohlool commented Apr 10, 2018 •

edited

Loading