Move CRDs to v1beta1 for 1.0.0-beta1

[pebrc]

Considering the (breaking) API revisions we have made and given that we want to express the beta level maturity of our CRDs we should move our resources to v1beta1 for 0.9.01.0.0-beta1.

[mario-mazo]

Are there any pointers on where to start? and what is the outcome, I would like to check this. But no idea where to start

[sebgl]

@mario-mazo thanks for your interest!
I think we'd like to wait a bit more before tackling this issue. 0.9 is not out yet, it's hard to know what 0.10 will look like in details for now.

[racevedoo]

How is this going? Is it still up for grabs? I'd like to take a look at it

[thbkrkr]

I think it's wise to do this after the merge of #1723, because it affects a lot of files.

[pebrc]

Two issues we have to consider here:

Do we need a minimal form of backwards compatibility?

What I mean by that: should it be possible for people running a previous version of ECK to install 1.0.0-beta1 into an existing cluster?

If the answer is to that question is yes then we have to ship both API versions (v1alpha1 and v1beta1) in 1.0.0-beta1

• Include both models in the code
• Fix the CRD generation (currently not working, not in use)
• Generate a CRD containing both version (using CRD multi-version support beta since v1.11 )
• Ensure that our controller version check filters out old versions
  • Why? What will happen is that controller runtime will serialise the old alpha ES clusters into beta model and try to reconcile them. See Bump CRD versions to v1beta1 #1782 (comment)

If the answer to "Do we need minimal backwards compatibility?" is no we should be aware of the consequences:

• ECK 1.0.0-beta1 will fail to start on k8s clusters which still have alpha ES clusters running (REST mapper for v1alpha1 is missing)
• Users will have to completely remove the old installation before being able to install 1.0.0-beta1 i.e. delete all ES clusters/Kibanas/APMs and delete the CRDs (see why below in kubectl section)
• It is unclear how users coming from operatorhub.io would do an upgrade
  • I believe old CRDs are left in place even when the subscription is removed
  • I assume that the experience will be similar and the new version would not boot up

Finally there is the question how will we retire v1alpha1?

• we/the user would have to either delete all old instances
• or we convert them to the new API model using a conversion webhook (beta in k8s v1.15 IIRC)
• we cannot do that on older k8s versions

kubectl apply -f crd.yaml and annotation max length

This second issue became apparent when experimenting with multiple versions. When using kubectl apply kubectl create the last-applied-configuration annotation that contains the previous version of the resource

• with a multi-version CRD this becomes too metadata.annotations: Too long: must have at most 262144 characters
• we can strip the descriptions from the CRDs to stay below the limit
• but it seems when maintaining more than on API version we might run into this again (say we have to support 3 versions temporarily)

[pebrc]

Regarding the kubectl apply issue, there are various options:

• use the trivialVersions flag in controller-gen (ES CRD 1K, no validation in api server)

Produce CRDs that work back to Kubernetes 1.11 (no version conversion)

• use maxDescLen=0 to strip out all the descriptions (ES CRD will be 222K instead of 700K)
  CRD_OPTIONS ?= "crd:trivialVersions=true"