rest.Config.NegotiatedSerializer causes watches to be created with the wrong scheme

[howardjohn]

We are seeing errors like watch of *v1alpha2.Gateway ended with: an error on the server ("unable to decode an event from the watch stream: unable to decode watch event: no kind \"Gateway\" is registered for version \"gateway.networking.k8s.io/v1alpha2\" in scheme \"pkg/runtime/scheme.go:100\"") has prevented the request from succeeding?

But...

• It works with a standard client-go informer
• I turned on debug logs, got the exact watch request to the api-server, and run it with kubectl get --raw and it works
• The controller actually works - its reading the resource its failing to watch. It almost seems like it can list but not watch?

After a lot of debugging, the root cause was setting rest.Config.NegotiatedSerializer in the rest config passed to the manager creation.

I am not sure if this is a bug or user error. If its a bug, the root cause is here:

controller-runtime/pkg/client/apiutil/apimachinery.go

Line 161 in 8f9d0df

if cfg.NegotiatedSerializer == nil {

The reason we have it set is we copy+pasted from https://github.com/kubernetes/kubectl/blob/652881798563c00c1895ded6ced819030bfaa4d7/pkg/cmd/util/kubectl_match_version.go#L115. I am not sure if there is a valid reason beyond that.