envtest with Kubernetes 1.20

[cbandy]

There is a kubebuilder-tools-1.20.2 since kubernetes-sigs/kubebuilder@3147a65, but as pointed out in kubernetes-sigs/kubebuilder#1902 this version has stronger opinions about --insecure flags. See kubernetes/kubernetes@cfc2b33.

When I try to use controller-runtime@v0.8.1 with kubebuilder-tools-1.20.2, the control plane does not start:

Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --insecure-bind-address has been deprecated, This flag has no effect now and will be removed in v1.24.
Error: invalid port value 56497: only zero is allowed
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --insecure-bind-address has been deprecated, This flag has no effect now and will be removed in v1.24.
Error: invalid port value 56497: only zero is allowed
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --insecure-bind-address has been deprecated, This flag has no effect now and will be removed in v1.24.
Error: invalid port value 56497: only zero is allowed
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --insecure-bind-address has been deprecated, This flag has no effect now and will be removed in v1.24.
Error: invalid port value 56497: only zero is allowed
Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Flag --insecure-bind-address has been deprecated, This flag has no effect now and will be removed in v1.24.
Error: invalid port value 56497: only zero is allowed

    test.go:37: failed to start the controlplane. retried 5 times: timeout waiting for process kube-apiserver to start

[alvaroaleman]

Yeah, getting this fixed and not using insecure-port in envtest would be great. In addition to working with newer kube versions, it would also allow us to write tests that require rbac and not be implemented via the allow-all authorized that is used for the insecure port, xref #1346 (comment)

/help

[k8s-ci-robot]

@alvaroaleman:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

Yeah, getting this fixed and not using insecure-port in envtest would be great. In addition to working with newer kube versions, it would also allow us to write tests that require rbac and not be implemented via the allow-all authorized that is used for the insecure port, xref #1346 (comment)

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[DirectXMan12]

I'm poking at this now

/assign @DirectXMan12

[alvaroaleman]

@DirectXMan12 WDYT about merging #984 and building on top of that?

[DirectXMan12]

lemme take a look

[DirectXMan12]

hmmm... insecure just stops working in 1.20, and secure should work across all the versions we support, so I'd lean towards making stuff work by default, and making an optional switch to force insecure -- something like

type Environment struct {
  // ...
  
  // ForceInsecure forces communication to the API server to occur over the insecure port.
  // This will only work on API servers 1.19 and below.  It's recommended that you use
  // the default and communicate over the secure port.
  //
  // Deprecated: Only works with Kubernetes 1.19 and below, will be removed once those age out.
  ForceInsecure bool
}

[DirectXMan12]

I think we can make this transparent by populating the rest.Config correctly and doing a bit more setup. It's still possibly a breaking change if people are constructing connections manually -- might want heavy beta testing for this one.

However, the alternative of forcing people to manually turn this on for the future or use different fields seems less ideal to me, I think.

[DirectXMan12]

I'll poke around a bit. Lemme see what it actually looks like in practice

[alvaroaleman]

Great! Only one wish: lets please not use the AllowAll authorizer, because it makes it impossible to write tests where you need the apiserver to respond with a 403 for some apis only. We can still by default return a kubeconfig that has global admin perms, that still leaves the possibility of using impersonation for the 403 case.

[DirectXMan12]

ack 👍 yeah, admin for a default case, plus a helper to produce non-admin users.