Allow to set Service LoadBalancer IP

[mazzy89]

Description:
At the current state it is not possible to set to the created Service LoadBalancer an user static IP address. The property is not exposed.

There are cases in various cloud providers like GCP where it is possibile to create a static IP address and reference it around.

There is a TODO in the EnvoyProxy ServiceSpec.

[mazzy89]

I don't think it should be hard to set so I can wire a PR.

[arkodg]

it looks like loadBalancerIP is deprecated https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
looking at the GKE docs https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer-parameters can you achieve the same by using a networking.gke.io/internal-load-balancer-subnet annotation to specify a /32 subnet (10.11.12.13/32)?

[mazzy89]

Right. I did not know it was going to be deprecated. Fair enough. Then the way to go is as recommend to implement the option using the Gateway API. For instance the GKE gateway allow to set an option to define the static ip address. Unfortunately that service annotation pointed by you does not cover this use case. I need to set up a public static IP address.

[arkodg]

yes you could always use GKE gateway but you'd be missing out on the extensive features in Envoy Gateway :)

even Envoy Gateway supports Gateway.Spec.Addresses but it doesnt affect the Cloud Loabalancer

curious why networking.gke.io/internal-load-balancer-subnet won't work, my guess is, if you submit subnet to be 1 IP (/32 range for IPv4) , then the service must be assigned that IP

[mazzy89]

As said above I have a public IP v4. That annotation is internal subnet. Doesn't not relate at all with public IP addresses.

[arkodg]

ah yes, thanks for the clarification

[mazzy89]

Then I think the way to go would be as the approach proposed done by Google https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#use_a_named_ip_address where the IP address would be referenced as a named IP address. However this would be a GCP only approach. what do you think?

[arkodg]

I dont think above approach will work, the Gateway defined is tied to a GKE GatewayClass gke-l7-rilb which is tied to a GKE controller which is probably doing some magic under the hood to stitch together a path from the Google LB to the Loadbalancer Gateway service

[mazzy89]

Sure. The logic there is tight to GKE but so would be also ours. I mean the logic in the envoy gateway controller would be: get the IP address from GCP using the named address, assign this IP to the Service LoadBalancer. This last part I'm sure it should be vendor agnostic and Kubernetes sig-network group has some idea how to achieve that. I haven't looked at it yet.

[arkodg]

Envoy Gateway supports the ability to set an address to the Gateway https://gateway.envoyproxy.io/v0.5.0/user/gateway-address.html which translates to Service.Spec.ExternalIPs , so If we set Gateway.Spec.Addresses to above NamedAddress, traffic will def enter Envoy Proxy pods, but the question is who is guaranteeing that traffic with DST IP set to NamedAddress is actually entering the k8s cluster

[mazzy89]

We would still need to implement in the envoy gateway controller gcp vendor logic to make sure that in the provider THAT named ip addr is set to the provider LB.

[AliceProxy]

+1 IMO we definitely need a way to support configuring this on a per-gateway basis, even if it is only a temporary solution since even though the field is deprecated, it is still the main way that GKE users assign IPs to LB services.

cc @LanceEa if you want to add any other comments or ideas/suggestions to this conversation

[LanceEa]

@mazzy89 - Thanks for raising this and @AliceProxy thanks for nudging me 😄. I have had notes on this subject and plans to write-up a proposal for a few weeks now. It would be great if there was a one size fits all for this but as always the devil is in the details.

tl;dr; My proposal is that we add a field to the EnvoyProxy config EnvoyProxy.spec.provider.kubernetes.envoyService.loadBalancerIP which will populate the Service this will ensure we can support the two most common ways of obtaining a Static IP address by either setting annotations or setting this field.

Pros:

• supported by both Azure and GKE (only supported way to get external IP AFAIK)
• K8s isn't going to be removing it anytime soon so no worries here (see analysis below)
• Keeps Service config centralized
• Limits GKE users blast radius when bad config is fed to Envoy Gateway due to tearing down infrastructure (a separate issue I have a TODO to do a write-up on)

Cons:

• It doesn't support IPV6 so it comes with caveats
• Limits support for multiple Gateways that are provisioned per EG instance since the EnvoyProxy config is shared* and only attaches to the GatewayClass today - (I think there is an open issue to address this though)

• I believe this is an issue when setting the static IP via an annotation as well so its not limited to this new field.

Why a static IP addresses is preferred?

I think this is well known but thought I would add anecdote that is driving my interest in this feature. I recently caused an outage on some internal services that are running through EG on GKE. Since EG was creating the Service it would create the Public external IP address which I then update DNS to resolve to it.

I then submitted some bad config which to my surprise caused EG to delete all my infrastructure. Which was then compounded because when I did my roll back in gitops the IP address had changed so I needed to then go update DNS again to get all the services back up an running.

This led me down the rabbit hole of research which is below 😄.

K8s Deprecation

Although the docs state it is deprecated, it seems that this is a loose "deprecation" and more of a nudge/caveat driven deprecation.

• Initial Deprecation PR and discussion: Deprecate Service.Spec.LoadBalancerIP kubernetes/kubernetes#107235
• Pull back PR stating its not being removed but discouraged due to the IPV6 limitation: Modify the API docs to not say that Service.Spec.LoadBalancerIP will be deleted kubernetes/kubernetes#117051

K8s Service docs

As mentioned above but just copying from my research notes:

https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer

• has deprecated the loadBalancerIP and says it will be removed in the future. I think the key reason for this
  is probably due to the lack of IPV6 support with this field.
• ExternalIPs seems to be only for exposing actual Node IP addresses which isn't the typical setup for most cloud installs.
  This seems AFAIK, targeted more to the kind and bare metal setups which EG does support today with the Gateway-API as mentioned above

Azure AKS Support

Azure supports both the loadBalancerIP address and annotations as seen here: https://learn.microsoft.com/en-us/azure/aks/load-balancer-standard#specify-the-load-balancer-ip-address

Google GKE Support

GKE only supports the loadBalancerIP for setting a static IP address at this time. I tested externalIPs and it was rejected
by the admission controller. The following document outlines reasoning for no support: https://cloud.google.com/knowledge/kb/unable-to-reach-services-with-externalips-and-externaltrafficpolicy-local-000007947

I also saw this in stackoverflow where it appears to be a GCP employee commenting (I'm guessing)
https://stackoverflow.com/questions/73750700/what-is-the-replacement-for-the-deprecated-loadbalancerip-attribute-in-services

AWS Support

Legacy K8s in-tree CloudProvider

It looks like the in-tree provider is removed in K8s 1.27+ and seems AWS will force the usage of its own controller when
looking at the release/v1.27 branch.

When looking at the legacy in-tree here: https://github.com/kubernetes/legacy-cloud-providers/blob/b9cd095f9783c9c9d9e762579cbeb6e28c4d1e80/aws/aws.go#L4018. It appears that it supports annotations but maybe has limited support for loadBalancerIP dependingon the AWS LB (nlb, elb, alb).

AWS LB Controller

Supports lots of annotations and is probably the recommended AWS method going forward as the in-cluster providers are phased out.
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.5/guide/service/annotations/

[haq204]

Limits support for multiple Gateways that are provisioned per EG instance since the EnvoyProxy config is shared* and only attaches to the GatewayClass today - (I think there is an open issue to address this though)

yea there's a GEP to add parametersRef to gateway to configure per-gateway infrastructure. Once that's in place, we can start attaching EnvoyProxy on Gateways which would be pretty useful here.

I could even make an argument that staticIP should be a first class field for gateway infrastructure

[arkodg]

thanks for the details @LanceEa, based on the fact that the field is not going to be removed any time soon, im okay with EG adding support for it.

reg outage due to DNS record going stale, recommend External DNS + Gateway API integration :)
https://gateway.envoyproxy.io/v0.5.0/user/tls-cert-manager.html#using-externaldns