Address "Built-in Defaulting" KEP feedback #2082

apelisse · 2020-10-06T21:50:48Z

cc @sttts @liggitt @DirectXMan12 @jpbetz

liggitt

KEP needs approvers listed as well

keps/sig-api-machinery/1929-built-in-default/README.md

liggitt · 2020-10-06T22:37:35Z

keps/sig-api-machinery/1929-built-in-default/README.md

@@ -484,27 +501,27 @@ the existing defaulting functions
 2. Update kube-openapi to parse the `+default` marker and include it in
 the OpenAPI definition.
 3. Implement the new CRD semantics that clears `null` values from
-objects when the `x-kubernetes-remove-nulls` flag is present and true at
-the top-level (or in the CRD).
+objects when nullable is unset or false.


Implement the new CRD semantics for properties with nullable unset or false (which previously would have resulted in a validation error that null is not a valid value):

properties and list items with a default have the default applied, replacing the explicit null value

list items without a default let the explicit null value remain, resulting in a validation error (matches current behavior)

properties without a default have the explicit null value removed

liggitt · 2020-10-06T22:38:38Z

keps/sig-api-machinery/1929-built-in-default/README.md

+We propose to start removing fields with a null value, when the openapi
+nullable flag is missing or false. We assume that existing APIs that use


maybe tweak the ordering to clarify explicit null values will either be removed or defaulted. saying they'll be removed here, then defaulted later is hard to follow

keps/sig-api-machinery/1929-built-in-default/README.md

liggitt · 2020-10-07T01:58:02Z

keps/sig-api-machinery/1929-built-in-default/README.md

 ```

-Note that the generated code for built-in will look like this:
+Note that a simplified version of the generated code for built-ins would look like this:


give an example of the planned generated code for a complex object as well

That's not exactly what you asked, but I think it has the same purpose

keps/sig-api-machinery/1929-built-in-default/README.md

sttts · 2020-10-07T10:21:04Z

keps/sig-api-machinery/1929-built-in-default/README.md

-the objects would technically be invalid.
+For CRDs, invalid zero-values are omitted, otherwise the objects would
+technically be invalid. We can safely default these field if they are
+unspecified.


what does "are omitted" mean? I cannot follow.

Are you saying we are adding a validation step before defaulting for all fields that have zero value?

Note that being invalid is not an easy property to check.

Example:

type: string maxLength: 3 pattern: ^....$

or

type: string allOf: - minLength: 5 - maxLength: 4

In other words, you have to apply the full go-openapi/validator.

I'm not planning on changing any of the validation. If we allow empty strings today, we'll continue allowing empty strings with this proposal, but they won't be defaulted. Defaults will only be applied to unspecified or null values on non-nullable fields.

I agree that this doesn't solve all the problems that we would like to solve.

One such problem is for example, container protocol associative key. If someone embeds that container port structure in a CRD, and the "" string is passed for the protocol, it won't be defaulted and will do the wrong thing when it comes to using the associative key (CRD will think the key is [$port, ""], but when the pod template is applied, the empty string will get defaulted to TCP and the key will become [$port, TCP]).

To be fair, the fact that we can't differentiate between empty strings and missing strings for built-in type is not addressed by this proposal and still needs to be solved.

We can improve incrementally on that in a backward compatible way since this doesn't close the door to improve the zero-value semantics later, if we decide that it is worth it.

IMO this is a blocker for the whole enhancement. It clearly states in the goals:

we will make the behavior equivalent between built-in types and CRDs

This goal is not reached.

x-kubernetes-prune-zero: true

How about something like that? And kubebuilder automatically adding that.

Yeah, that's the obvious solution, which would be generated on "omitEmpty" non-pointers I assume. I like that this is not an option to users.

Am fine with something more advanced in kubebuilder that knows the Go typesystem and when json.Marshal would apply omitempty and when it doesn't. In the CRD schema API we only have to provide the primitives such that kubebuilder can apply them accordingly.

keps/sig-api-machinery/1929-built-in-default/README.md

sttts · 2020-10-12T09:48:51Z

#2082 (comment) is serious to move forward. Not having a plan about a restriction of the approach which makes not reach the stated goal is a nogo.

apelisse · 2020-10-16T16:37:42Z

I think we're good with this, can we merge this?

keps/sig-api-machinery/1929-built-in-default/README.md

liggitt · 2020-10-19T00:07:57Z

Found a few things to clarify and a few examples to fixup, mostly lgtm otherwise. Only real question I had was about the "omitempty on non-pointer scalar" requirement when the zero-value is set as an explicit default (#2082 (comment))

keps/sig-api-machinery/1929-built-in-default/README.md

sttts · 2020-10-19T12:53:07Z

keps/sig-api-machinery/1929-built-in-default/README.md

@@ -483,28 +512,34 @@ the additional defaulting functions, automatically call them prior to
 the existing defaulting functions
 2. Update kube-openapi to parse the `+default` marker and include it in


I miss the discussion and conclusion about publishing the OpenAPI defaults. For CRDs today we do not publish them intentionally. I suggest to do the same here.

For CRDs today we do not publish them intentionally.

I didn't know that. I don't really understand it, but I guess I'm fine if we do the same. As long as server-side apply is able to read them since that's one of the reason why we're building that mechanism.

SSA will see the full schema. This happens as part of the OpenAPI v2 conversion.

Have you added a clarification anywhere?

No, not yet, I'm not super satisfied to hide them. it means we can't generate a good "openapi" example to test the behavior in kube-openapi without re-adding the defaults, it means client can't merge properly using listmapkey because they don't have the default either. The more expressive the schema the better, we should keep the defaults, and we should probably do so for CRDs too.

See my text in the gdoc. I still think we have to describe our plan.

keps/sig-api-machinery/1929-built-in-default/README.md

sttts · 2020-10-22T08:56:55Z

keps/sig-api-machinery/1929-built-in-default/README.md

+nullable are happy with null values. Also, people can remove the
+nullable attribute if they want null values to start being removed. This
+will help aligning CRDs and built-in types (there are no ways to set
+nullable to true for built-in types).


Maybe worth a short note:

We will remove null fields also from map[string]Ts, i.e. for type that have additionalProperties in their schema instead of properties.

sttts · 2020-10-22T09:01:03Z

keps/sig-api-machinery/1929-built-in-default/README.md

-     // +default=default-name
-     Name string
+     // +default="default-name"
+     Name string `json:"name,omitempty"`


this is no example for the cases above:

"" is valid

the default is not the zero value.

How do you know "" is valid? The comment reads "empty string is not a valid string", we just can't differentiate between unset and invalid.

Why does it matter whether it is valid or not? Why not the other cases? This is not obvious.

sttts · 2020-10-22T09:04:56Z

keps/sig-api-machinery/1929-built-in-default/README.md

+Non-pointer scalar fields that have a default must be omitempty (unless
+their default is the zero-value). Kubebuilder and built-in defaulting
+generators should both require this.
+
 This would generate the following OpenAPI:


where is the golang source for the schema below?

Ah yes, the name doesn't match, good catch

sttts · 2020-10-22T09:09:56Z

Am still not happy with the examples section. There are still a number of typos and missing cases. I would also like to see a clear distinction between the proposed

pruning behaviour
defaulting behaviour
kubebuilder/defaulter-gen changes.
examples.

Especially the lack of separation of 1-3 vs. 4 makes it read very handwavy.

sttts · 2020-10-22T18:39:03Z

keps/sig-api-machinery/1929-built-in-default/README.md


-     // This field is defaulted to 0 with no defaulter.
+     // This field would be defaulted to 0 without a default.


Why would it be default to 0 without a default? What about non-omitempty cases in general?

apelisse · 2020-10-22T22:10:46Z

keps/sig-api-machinery/1929-built-in-default/README.md


-     // This field is defaulted to 0 with no defaulter.
-     // +default=0
+     // This field would be defaulted to 0 without a default.


That defaulted below looks a little weird now.

sttts · 2020-10-23T05:44:45Z

keps/sig-api-machinery/1929-built-in-default/README.md

 built-in that would behave the same way.

-### CRD Normalization
+We will depend on tooling (defaulter-gen and kubebuilder) to enforce the
+new type requirements (via linting, warnings or errors), to implicitly


nit: linting and warnings was meant to be the same

sttts · 2020-10-23T05:46:04Z

keps/sig-api-machinery/1929-built-in-default/README.md

+   +default marker
+2. we forbid or warn (liniting) about omitempty on non-pointer structs
+3. we require omitempty on scalars with default different from the
+   zero-value.


see my 4th rule in the gdoc

sttts · 2020-10-23T05:47:58Z

keps/sig-api-machinery/1929-built-in-default/README.md

     // +default="default-name"
-     Invalid string `json:"name,omitempty"`
+     Name string `json:"name,omitempty"`


json:"name,omitempty" by rule 3

sttts · 2020-10-23T05:48:41Z

keps/sig-api-machinery/1929-built-in-default/README.md

-`omitempty`. Non-pointer scalar fields that have a default must be
-omitempty (unless their default is the zero-value). Kubebuilder and
-built-in defaulting generators should both require this.
+`omitempty`. On the other hand, a manifest in YAML


a manifest in JSON

sttts · 2020-10-23T05:49:01Z

keps/sig-api-machinery/1929-built-in-default/README.md

+
+{"entry": {"invalid": "", "number": 0}}
+
+converted to JSON directly and sent to the server would result in


remove "converted to JSON direction and"

sttts · 2020-10-23T05:49:14Z

keps/sig-api-machinery/1929-built-in-default/README.md

+// +default="apple"
+type Item string
+```
+


flesh this one out, same for string maps

sttts · 2020-10-23T05:55:21Z

keps/sig-api-machinery/1929-built-in-default/README.md


 As described by @sttts on many occasions, the immutability is blocked
 because we don't know how to distinguish between null and
 unspecified. We believe that this mechanism will be able to unblock it.

+### Changes to kubebuilder/default-gen


@DirectXMan12 please take a look

sttts · 2020-10-23T18:24:40Z

keps/sig-api-machinery/1929-built-in-default/README.md

-   zero-value.
+zero-value
+4. we auto-generate `default: <zero-value>` for scalars without
+omitempty and forbit the `+default` marker


sttts · 2020-10-23T18:25:47Z

keps/sig-api-machinery/1929-built-in-default/README.md

+>>> default('{mapping: {foo: null, bar: apple}')
+{"mapping": {"foo": "banana", "bar": "apple"}
+```
+


maybe another case with null and no default => removal

sttts · 2020-10-23T18:26:08Z

keps/sig-api-machinery/1929-built-in-default/README.md

+>>> default('{list: [null, foo]}')
+{"list": ["apple", "foo"]}
+```
+


add case with null and no default => NO removal

sttts · 2020-10-23T18:27:32Z

Only two nits left: examples for array and string map with null and no default

Other than that: LGTM

apelisse · 2020-10-23T18:49:43Z

Added the example and rebased!

lavalamp · 2020-10-23T18:57:07Z

keps/sig-api-machinery/1929-built-in-default/README.md

+  properties:
+    mapping:
+      type: object
+	  additionalProperties:


Co-authored-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

lavalamp · 2020-10-23T19:02:24Z

/approve

Maybe @DirectXMan12 can look at the section @sttts tagged them in and do the final LGTM?

k8s-ci-robot · 2020-10-23T19:02:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apelisse, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-api-machinery/OWNERS~~ [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

DirectXMan12

couple of nits inline, but consider this a soft lgtm, expecially in the part I was directly tagged in

/lgtm

DirectXMan12 · 2020-10-28T21:54:50Z

keps/sig-api-machinery/1929-built-in-default/README.md

      type: string
-      default: default-name
+      default: "default-name"


NIT: don't actually need the quotes here -- it's the same either way ;-)

DirectXMan12 · 2020-10-28T21:57:11Z

keps/sig-api-machinery/1929-built-in-default/README.md

+sent to the server would result in different behaviour for built-ins
+
+```python
+>>> default('{name: ""}')


NIT: it's a bit weird that the example object changes when it looks like you're going to keep talking about the above invalid entry.

DirectXMan12 · 2020-10-28T21:57:37Z

keps/sig-api-machinery/1929-built-in-default/README.md

+and CRDs:
+
+```python
+>>> default('{name:"", number:0}')


number vs defaulted?

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 6, 2020

k8s-ci-robot requested review from deads2k and lavalamp October 6, 2020 21:51

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Oct 6, 2020

liggitt reviewed Oct 6, 2020

View reviewed changes

liggitt reviewed Oct 7, 2020

View reviewed changes

sttts reviewed Oct 7, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 7, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 7, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

liggitt reviewed Oct 19, 2020

View reviewed changes

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Outdated Show resolved Hide resolved

sttts reviewed Oct 19, 2020

View reviewed changes

keps/sig-api-machinery/1929-built-in-default/README.md Show resolved Hide resolved

apelisse force-pushed the fix-defaulting-kep branch from 06a2b1d to 37f578d Compare October 21, 2020 17:26

sttts reviewed Oct 22, 2020

View reviewed changes

apelisse commented Oct 22, 2020

View reviewed changes

sttts reviewed Oct 23, 2020

View reviewed changes

apelisse force-pushed the fix-defaulting-kep branch 2 times, most recently from f32d5b0 to d3906e5 Compare October 23, 2020 18:46

lavalamp reviewed Oct 23, 2020

View reviewed changes

Address post-merge feedback

e2baa04

Co-authored-by: Dr. Stefan Schimanski <stefan.schimanski@gmail.com>

apelisse force-pushed the fix-defaulting-kep branch from d3906e5 to e2baa04 Compare October 23, 2020 18:58

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 23, 2020

DirectXMan12 reviewed Oct 28, 2020

View reviewed changes

k8s-ci-robot assigned DirectXMan12 Oct 28, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 28, 2020

k8s-ci-robot merged commit 56f1ae1 into kubernetes:master Oct 28, 2020

k8s-ci-robot added this to the v1.20 milestone Oct 28, 2020

luxas mentioned this pull request Sep 16, 2021

Upgrade yaml.v2 to yaml.v3, improve testing and bugfixes kubernetes-sigs/yaml#61

Closed

		We propose to start removing fields with a null value, when the openapi
		nullable flag is missing or false. We assume that existing APIs that use

		@@ -483,28 +512,34 @@ the additional defaulting functions, automatically call them prior to
		the existing defaulting functions
		2. Update kube-openapi to parse the `+default` marker and include it in


		// This field is defaulted to 0 with no defaulter.
		// This field would be defaulted to 0 without a default.


		{"entry": {"invalid": "", "number": 0}}

		converted to JSON directly and sent to the server would result in

Address "Built-in Defaulting" KEP feedback #2082

Address "Built-in Defaulting" KEP feedback #2082

Conversation

apelisse commented Oct 6, 2020

liggitt left a comment

Choose a reason for hiding this comment

liggitt Oct 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts Oct 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

sttts commented Oct 12, 2020

apelisse commented Oct 16, 2020

liggitt commented Oct 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts Oct 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apelisse Oct 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented Oct 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts Oct 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented Oct 23, 2020

apelisse commented Oct 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lavalamp commented Oct 23, 2020

k8s-ci-robot commented Oct 23, 2020

DirectXMan12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Oct 6, 2020 •

edited

Loading

sttts Oct 7, 2020 •

edited

Loading

sttts Oct 12, 2020 •

edited

Loading

sttts Oct 22, 2020 •

edited

Loading

apelisse Oct 22, 2020 •

edited

Loading

sttts commented Oct 22, 2020 •

edited

Loading

sttts Oct 23, 2020 •

edited

Loading