Allocation Endpoint - Better load balancing for gRPC connection

[markmandel]

Is your feature request related to a problem? Please describe.

The Kubernetes load balancer doesn't load balance a single gRPC connection across multiple pods, so when doing Allocations via the gRPC endpoint, a single client will only ever connect and use a single allocation Pod.

Describe the solution you'd like

Have the system load balance across each of the allocation pods, without requiring the external integration team (i.e. the end user) to do extra work.

Some research:
1. https://grpc.io/blog/grpc-load-balancing/
1. https://blog.nobugware.com/post/2019/kubernetes_mesh_network_load_balancing_grpc_services/
1. https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/
1. https://itnext.io/proxyless-grpc-load-balancing-in-kubernetes-ca1a4797b742
1. https://cloud.google.com/solutions/exposing-grpc-services-on-gke-using-envoy-proxy

The Envoy based solution from Google Cloud looks to me to be the most viable.

The potentially interesting part would be that the TLS certificates would then be handled by Envoy rather than the Allocation endpoint itself.

Describe alternatives you've considered

Leave things as is. Allocation seems to be performing reasonably well (or at least it should do once #1863 is merged)

Additional context

Initially discussed here: #1867 (comment)

[markmandel]

@pooneh-m would love your thoughts.

[markmandel]

This would now have to cover both gRPC and HTTP endpoints - as we handle both now in the 1.11.0 RC that came out yesterday.

[roberthbailey]

@ilkercelikyilmaz FYI

[github-actions]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[github-actions]

This issue is marked as obsolete due to inactivity for last 60 days. To avoid issue getting closed in next 30 days, please add a comment or add 'awaiting-maintainer' label. Thank you for your contributions