Upgrade Kubebuilder to v2, controller-tools and controller-runtime to 0.2

[nkvoll]

When v2 / 0.2 of the above is released, we should upgrade our dependency.

Relevant Kubebuilder project board: https://github.com/kubernetes-sigs/kubebuilder/projects/6

[anyasabo]

This will be a little bit involved, I'm starting on it but we may want to split this up once I have a better idea of what it entails.

One issue we're running into is that we use multiple groups (e.g. kibana.k8s.elastic.co, elasticsearch.k8s.elastic.co) which takes some extra work in kubebuilder v2:
https://book.kubebuilder.io/migration/multi-group.html
and
https://kubernetes.slack.com/archives/CAR30FCJZ/p1565293673264500

Another is that webhooks no longer auto register which we will need to decide how to deal with:
https://book.kubebuilder.io/migration/v1vsv2.html

They seem to be pushing for cert manager by default which might be a stance we want to consider.

[pebrc]

This will be a little bit involved

Agreed. I think we need to do some upfront planning in this case before we get started.

They seem to be pushing for cert manager by default which might be a stance we want to consider.

I think while they are pushing for cert-manager given how convenient it is to just inject the ca certs into the webhook via an annotation or into the server via kustomize, I also think that we, unfortunately, need to support use cases without cert manager as well.

The way I understand it, there are two places where certificates need to be injected: the admission webhook resource and the webhook server. I think in both cases it should be possible to implement a fallback if cert-manager is not in the mix, which we cannot expect it to be all the time. We should then just generate our own self-signed certificates as we are already doing in many places.

It could even mean that this is an opportunity to replace our internal cert generation optionally via some flag with an external tool like cert-manager (maybe with the exception of the transport certs with have use the other name for the node name in the SAN extension)

[anyasabo]

Agreed. I think we need to do some upfront planning in this case before we get started.

I started taking a run at it to get an idea of what all goes into it. I can back off if someone else has already done a spike. Just looking at the kubebuilder v2 project board it looks like it's more or less ready for us to migrate.

there are two places where certificates need to be injected: the admission webhook resource and the webhook server.

that's my understanding too.

We should then just generate our own self-signed certificates as we are already doing in many places.

I think this is a little more convoluted because the webhook configuration needs the CA that signed the webhook's certificate. I'm not sure there's an easy way to distribute something people can just kubectl apply. It seems like we'd need to provide some kind of script for people to generate certs and update their operator/webhook config.

It could even mean that this is an opportunity to replace our internal cert generation optionally via some flag with an external tool like cert-manager (maybe with the exception of the transport certs with have use the other name for the node name in the SAN extension)

Yup that too. I know we had resisted having a dependency on cert-manager in the past but I think it might be okay. Obv we would want people to supply their own certs somehow too, but I think it's an idea worth exploring.

[pebrc]

think this is a little more convoluted because the webhook configuration needs the CA that signed the webhook's certificate. I'm not sure there's an easy way to distribute something people can just kubectl apply.

Yes, I think you are right, the problem here is that cert-manager injects the certificates into the resource based on a label kubebuilder attaches. To support a similar workflow without cert-manager we would need to give the operator permissions to edit admission/mutating webhooks and then identify the webhooks created by kubebuilder or see if we can generate webhooks at installation time and inject certificates there (basically rebuilding what the pre-0.2 kubebuilder did)

For the webhook server things are simpler as we just need to create a TLS secret called webhook-server-cert

[pebrc]

closed via #1723