Implement KEP-1860 in AntreaProxy: Make Kubernetes aware of the LoadBalancer behavior

[hongliangl]

Describe what you are trying to do

In Kubernetes 1.29, a new field LoadBalancerIPMode is added to in LoadBalancerIngress. This is for KEP-1860. This field is used to specify how the load-balancer IP behaves of a Service. According to the explanation, the value can be:

• LoadBalancerIPModeVIP, behaves the same as current implementation.
  1. client opens a socket to the IP: {src = client_ip, dst = lb_ip:service_port} (Refer to Make Kubernetes aware of the LoadBalancer behaviour kubernetes/kubernetes#118895 (comment))
  2. routing forwards that to the node: {src = client_ip, dst = lb_ip:service_port}
  3. node receives traffic and DNATs to pod IP
  4. pod accepts the connect
• LoadBalancerIPModeProxy, the related Service traffic is sent to external loadBalancer.
  1. client opens a socket to the IP: {src = client_ip, dst = lb_ip:service_port}
  2. proxy accepts the connect
  3. proxy opens a socket to the node or pod: {src = proxy_ip, dst = pod_ip:pod_port}
  4. (optional) node receives traffic on NodePort and DNATs to pod IP
  5. pod accepts the connect
• nil, the same as LoadBalancerIPModeVIP.

Currently, we have an option antreaProxy.proxyLoadBalancerIPs which is used to decide whether to process LoadBalancer traffic on K8s Node locally or send the traffic to external loadBalancer. This option take affects on all existing LoadBalancer Services. By introducing KEP-1860 in AntreaProxy, users can have more fine-grained control over the LoadBalancer behavior.

Dependencies:

• GA version of k8s.io/api 0.28

Cases:

• AntreaProxy option proxyLoadBalancerIPs: true, Service LoadBalancerIPMode: LoadBalancerIPModeVIP, traffic is processed in AntreaProxy.
• AntreaProxy option proxyLoadBalancerIPs: true, Service LoadBalancerIPMode: LoadBalancerIPModeProxy ?
• AntreaProxy option proxyLoadBalancerIPs: false, Service LoadBalancerIPMode: LoadBalancerIPModeVIP, ?
• AntreaProxy option proxyLoadBalancerIPs: false, Service LoadBalancerIPMode: LoadBalancerIPModeProxy, traffic is processed in external loadBalancer.

[hongliangl]

@tnqn @antoninbas Could you help give some suggestions? Thanks a lot.

[antoninbas]

I think we have 2 options:

Option 1:
When proxyLoadBalancerIPs is true (default), always use the LoadBalancerIPMode behavior
When proxyLoadBalancerIPs is false, never process traffic in AntreaProxy

Option 2: leverage the fact that proxyLoadBalancerIPs is a pointer (*bool) in the config
When proxyLoadBalancerIPs is omitted (null), always use the LoadBalancerIPMode behavior
When proxyLoadBalancerIPs is true, always process traffic in AntreaProxy
When proxyLoadBalancerIPs is false, never process traffic in AntreaProxy

IMO, either option is acceptable from a usability and backwards-compatibility aspect. I have a slight preference for Option 1, as it is a bit simpler, but let's hear what @tnqn thinks. The ability of Option 2 to override LoadBalancerIPMode for all Services and always LB traffic in AntreaProxy doesn't seem very useful.

[tnqn]

Option 1 makes sense to me. It's easy to explain and understand. The true value could be explained as "proxy load balancer IPs when it's asked to"; the false value could be explained as "don't proxy load balancer IPs".

Option 2 may have a few issues:

1. Currently proxyLoadBalancerIPs has been set to true in config file, we will need to change it to nil.
2. LoadBalancerIPMode could also be nil before K8s 1.28, and when the feature is disabled. There would be too many combination (3*3) and defaulting processes.

[antoninbas]

@hongliangl sounds like the consensus is for option 1

[hongliangl]

Thanks a lot! Option 1 is easy to understand and �code.