charts: add ReferenceGrant permissions to Consul API Gateway ClusterRole

[mikemorris]

Changes proposed in this PR:

• Add ReferenceGrant permissions to Consul API Gateway ClusterRole, refs Rename files to reflect ReferencePolicy -> ReferenceGrant kubernetes-sigs/gateway-api#1188
• Doesn't remove ReferencePolicy yet, as we want to allow backwards compatibility until the Gateway API v0.6.0 release, refs Adding back ReferencePolicy with a note that it will be removed in v0.6.0 kubernetes-sigs/gateway-api#1205
• Refs Allow Consul API Gateway controller to read ReferencePolicy #1148

How I've tested this PR:

• Deployed Consul API Gateway from Helm chart from this branch and ran Gateway API conformance tests.

[INFO]  k8s/logger.go:30: consul-api-gateway-server.controller-runtime: Starting EventSource: controller=tcproute controllerGroup=gateway.networking.k8s.io controllerKind=TCPRoute info="Starting EventSource" source="kind source: *v1alpha2.ReferenceGrant"

How I expect reviewers to test this PR:

• Consul API Gateway is able to deploy successfully from this branch using the Helm chart, with the changes in gateway-api: update to v0.5.0 and v1beta1 resources consul-api-gateway#224 to watch ReferenceGrant objects. Without this change, the Gateway controller falls into a crash loop with the following error:

[ERROR] k8s/logger.go:35: consul-api-gateway-server.kubernetes-client: pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1alpha2.ReferenceGrant: failed to list *v1alpha2.ReferenceGrant: referencegrants.gateway.networking.k8s.io is forbidden: User "system:serviceaccount:consul:consul-consul-api-gateway-controller" cannot list resource "referencegrants" in API group "gateway.networking.k8s.io" at the cluster scope

Checklist:

• Tests added
• CHANGELOG entry added

  HashiCorp engineers only, community PRs should not add a changelog entry.
  Entries should use present tense (e.g. Add support for...)

[nathancoleman]

We'll definitely want a release note for this. If I'm not mistaken, anyone having apiGateway.enabled=true will have to update their gateway CRDs prior to upgrading to the release of consul-k8s containing this change.

[nathancoleman]

Should we so far as to include the command? If I wasn't a core dev on API Gateway, I feel like it'd be hard for me to determine from the release note here what exactly I need to do.

$ kubectl apply --kustomize "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=v0.5.0-rc1"

[mikemorris]

🤔 I think not, because the preferred way to install would be to install our CRDs, which include the necessary upstream ones (and installing directly like ^ will skip installing the deprecated ReferencePolicy CRD by default). However, it may be weird to land this before we have a tagged release that installs upstream CRDs including ReferenceGrant?

kubectl apply --kustomize "github.com/hashicorp/consul-api-gateway/config/crd?ref=v0.4.0"

[nathancoleman]

Our CRDs don't currently include the necessary ReferenceGrant CRD though and won't for a few weeks until we release API Gateway v0.4.0. There's a pretty good chance consul-k8s will cut a release before v0.4.0 exists. I'm mainly thinking of what consumers of the next version of consul-k8s will do in that interim period.

[mikemorris]

Yea, it's possible we just shouldn't merge this until the release cycle directly before our v0.4.0 release? Not quite sure the best way to handle this or communicate in support matrices.

[nathancoleman]

I was previously thinking it best to merge soon so we wouldn't have to ask for a special release of consul-k8s, but I agree with you. Releasing early seems like it will make the practitioner's life more difficult.