Support multiple GatewayClass

[arkodg]

Description:
Today, Envoy Gateway only supports a single GatewayClass. This decision was taken to simplify logic, limit permissioning and ensure sensitive material is isolated (across tenants) before dealing with multi tenancy (each GatewayClass is a tenant).

Our proposed alternate solution is to request users to create another Envoy Gateway controller with a different name in a the same cluster in a different namespace and link the newer GatewayClass to it

Keeping this issue open to gather feedback from users.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[wondersd]

Hi all,

I think it would be great if multiple GatewayClasses could be managed such that one controller is able to handle multiple "tenants" or even diverse Gateway instances within one logical "tenant". As it stands today, even with multiple EG controllers for each logical tenant, team X or environment Y, is still bound to only creating uniform Gateway instances within that tenant. If they wish to segregate internally for different purposes, e.g. internal vs external, or a critical service like an exposed database vs more forgiving lightweight api calls, they would have to install multiple EG controllers.

The resources allocated to it are dependent on what services are being proxied though it and are not 'automatic' as one may get via a cloud managed LB (like GKE's provided gateway api implementation mapping to GCE LB instances). If there is only ever one Gateway per GatewayClass then all is well. But trying to use multiple Gateway per GatewayClass one is not able to manage parameters of each envoy proxy cluster in accordance to what each separate Gateway is doing.

Having each "tenant" install a EG controller would require each tenant to have some higher level permissions to the cluster per each tenant, e.g. creating ClusterRole ClusterRoleBindings for the individualized EG controller or having to coordinate with a cluster administrator. In the case of many "tenants" or a high churn of "tenants" this is not easily manageable. This starts to lead down the road of installing a helm chart (or equivalent) per managed envoy proxy instance. Which, to me atleast, seems a bit underwhelming for the controller/operator model as its seen as a mechanism to provide "infrastructure as a service" within the k8s ecosystem. Here, providing a uniform service for managing envoy proxy clusters as defined by the gateway api spec.

I think it would be nice to either be able to specify some sort of glob pattern for watched controllerName like gateway.envoyproxy.io/*

and/or making

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway

a full CRD where those wanting to make GatewayClass would also need permissions to create/manage EnvoyGateway to map the new controllerName to watch for

Thank you all for working on this and would love to see this continue to get better!

[arkodg]

thanks for the feedback @wondersd !

• the first limitation that we can try and remove is attaching a EnvoyProxy resource to a Gateway, similar to what is done at the GatewayClass level today. This has been defined upstream https://gateway-api.sigs.k8s.io/geps/gep-1867/ and once the API moves to Experimental, we should be able to support it in the project. This will allow teams to have heterogeneous data plane (envoy proxy fleet) infrastructure while still being linked to the same GatewayClass

[wondersd]

@arkodg Thanks for the quick response! Any notion of how parameterRef on the Gateway will be used in EG? Would we expect that both could be used for a "default/template + override" or will EG shift to only expect parameterRefs attached to Gateway or something else?

For what its worth I think having multiple GatewayClasses could still be useful here even if one can customize each Gateway instance via parameterRefs. A cluster administrator could setup GatewayClass to abstract some of the details of the cluster. For example a GatewayClass instance called eg-internal on say an EKS cluster that specifically maps the envoy proxy service to be behind an NLB that is mapped to a set of subnets dedicated for such internal load balancers. Those making Gateway instances who are not the cluster administrators would be able to instance this class without needing to know the specifics of the cluster lower level setup while still being able to further customize each instance of said class for their needs.

I suppose one could fall back to multiple controller installs for the above, which in practice would probably be fine as the cluster administrator defining such GatewayClass instances would be the same role setting up the controllers in the first place.

Hope this is helpful, thanks for hearing me out!

[arkodg]

Thanks for the quick response! Any notion of how parameterRef on the Gateway will be used in EG? Would we expect that both could be used for a "default/template + override" or will EG shift to only expect parameterRefs attached to Gateway or something else?

• This was discussed in the community meeting and @AliceProxy, @kflynn and I agreed that this would be supported on both the GatewayClass & Gateway with the value in Gateway taking precedence if parametersRef was set on both the resources, which is also in line with the above GEP's recommendations.

regarding your request for multiple GatewayClasses being supported by EnvoyGateway, its def a valid use case that this project should eventually support and this issue should be used to track its completion.

thanks again for sharing your feedback !

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

will need to prioritize this one, since there are no concrete plans in upstream to support an arbitrary infra extension point such as parametersRef in the Gateway so supporting the pattern of multiple GatewayClass with each one attaching a unique EnvoyProxy inside the parametersRef seems like the only option for now

[arkodg]

if we implemented this, this should make #1851 redundant

[arkodg]

cc @cnvergence curious if you're interested in looking at this since its similar to the MergeGateways feature

[cnvergence]

@arkodg sounds good to me, I will pick it up once I will have some spare time

[arkodg]

awesome thanks @cnvergence

[Xunzhuo]

Hey @cnvergence, kindly ping for the status of the issue, are you still working on this : )

[cnvergence]

hey @Xunzhuo, sorry, did not move it forward lately, but now I will be able to have more time for it, so I would like to focus on this subject :)

[Xunzhuo]

Thank you @cnvergence, since it is an important feature in the next milestone, so just kindly ping you.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.