Generated Metadata breaks crd in 1.15

[skonto]

Details are described in kubernetes/kubernetes#80493. Initially I thought that this might be a k8s issue but it looks now like an incompatibility issue for controller-tools.

The problem is that the default preserveUnknownFields: true in combination with

  annotations:
    certmanager.k8s.io/inject-ca-from: test-operator-system/test-operator-serving-cert

and

conversion:
    strategy: Webhook
    webhookClientConfig:
      caBundle: Cg==
      service:
        name: test-operator-webhook-service
        namespace: test-operator-system
        path: /convert

create this error The CustomResourceDefinition "cronjobs.batch.tutorial.kubebuilder.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true

If you set preserveUnknownFields: false to make conversion work then immediately you get:

> The CustomResourceDefinition "testapplications.testoperator.k8s.io" is invalid: spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified

[skonto]

@DirectXMan12 am I missing something here? I dont see any docs about the above requirements in https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning btw, but code is there.

[skonto]

I verified that this works with 1.14.2 but not 1.15.{0,1}, tested with minikube and --feature-gates CustomResourceWebhookConversion=true for 1.14.2.

[liggitt]

preserveUnknownFields is a new field in 1.15. Anything setting preserveUnknownFields: false is opting into structural schema validation and must comply with all the structural requirements (see https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#specifying-a-structural-schema), including:

if metadata is specified, then only restrictions on metadata.name and metadata.generateName are allowed.

[liggitt]

And you are correct, structural schema is required to make use of webhook conversion

[DirectXMan12]

we should be no longer specifying validation for objectmeta in the latest versions of controller-tools, in part due to this issue.

[DirectXMan12]

can you retry with beta.5?

[ncdc]

@DirectXMan12 I'm seeing this too with beta.5. Still digging into it but wanted to report it here.

[ncdc]

Or rather, preserveUnknownFields is still true (the metadata validation is correctly switched to object).

[ncdc]

If I manually set preserveUnknownFields, I'm able to deploy everything.

[ncdc]

Aha, we need #281

[voelzmo]

Just found this as well when using kubebuilder on 1.15.3
I'm not sure how to fix this currently – can I do some manual changes until #281 finds its way?

[jgustie]

I found that v1beta1 generated CRDs that use IntOrString are problematic because of the use of x-kubernetes-int-or-string: this seems to force the use of structural schemas on the API side regardless of if preserveUnknownFields is explicitly set to "true" or left unset (and structural schema validation currently fails because of the inclusion of metadata definitions).

Manually removing the x-kubernetes-* lines from the v1beta1 generated CRDs allows me to deploy them.

[DirectXMan12]

@sttts is that intentional?

[liggitt]

yes, use of new features like x-kubernetes-... extensions requires structural schemas.

[muvaf]

Is there a flag or a way to disable use of x-kubernetes-* features during CRD generation? This is breaking 1.14 and lower versions and currently GKE default version is 1.13 and AWS EKS doesn't yet have support for 1.15

[DirectXMan12]

@muvaf that's probably a separate issue (just that client-side validation doesn't understand), but yeah, we should fix that too. In the short term, you can run with --validate=false on kubectl [xyz] config/crd/base. Can you file a separate issue please?

[muvaf]

@DirectXMan12 yeah, this seems to be the case. I tested with GKE 1.13.11 and kubectl 1.17 and it worked. Thanks!

Here is the separate issue #392

[advissor]

@muvaf @DirectXMan12 --validate=false I tried before with no success.
As far as I remember client/server compatibility version is +/- 1 minor.
So, 1.15 and 1.17 <> 1.13 are not compatible then , it this right?

https://kubernetes.io/docs/setup/release/version-skew-policy/

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frodopwns]

Ran into this in kube 1.16 docker-desktop. Generated project using Kubebuilder 2.3.

make install                                                                                                                                                         
~/go/bin/controller-gen "crd" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
The CustomResourceDefinition "cronjobs.batch.my.domain" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
make: *** [install] Error 1

[DirectXMan12]

@frodopwns the related issues should be solved. Can you confirm that you have the latest version of controller-tools installed?

[redradrat]

@frodopwns @DirectXMan12
yep, I'm running into this as well on controller-gen 0.3.0

is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
make: *** [Makefile:30: install] Error 1

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[rajasaur]

I have the same problem as @redradrat with Kubebuilder 2.3.1 and controller-gen v0.3.0 too (This was a project that was upgraded from Kubebuilder 2.2 and so changed the Makefile to update controller-gen from v0.2.4 to v0.3.0).

The CustomResourceDefinition "xx.yy.zz" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.