Update Egress API to support multiple Egress IPs and pools #4603

tnqn · 2023-02-06T09:30:37Z

Nodes in a cluster could reside in multiple subnets, but the egress IP needs to be routable in the underlay network and usually resides in the same subnet of the Node hosting it. Therefore, there may be a situation where no available nodes are eligible to host an egress IP address if all Nodes in a subnet are down, interrupting egress traffic of the workloads that use the IP.

As the first step of supporting the above scenario, the patch extends the Egress IP to support multiple Egress IPs and pools so that one Egress IP can be configured for each subnet, making Egress failover across the whole cluster possible.

Besides, it also adds a field, status.egressIP, to represent the effective Egress IP. When there is no eligible Node for any of the Egress IPs, the field will be empty.

codecov · 2023-02-06T10:11:32Z

Codecov Report

Merging #4603 (567e2b3) into main (272b56c) will decrease coverage by 3.12%.
The diff coverage is 56.66%.

❗ Current head 567e2b3 differs from pull request most recent head 58e97ac. Consider uploading reports for the commit 58e97ac to get more accurate results

@@            Coverage Diff             @@
##             main    #4603      +/-   ##
==========================================
- Coverage   71.99%   68.87%   -3.12%     
==========================================
  Files         406      403       -3     
  Lines       60856    59797    -1059     
==========================================
- Hits        43812    41187    -2625     
- Misses      14132    15773    +1641     
+ Partials     2912     2837      -75

Flag	Coverage Δ		*Carryforward flag
e2e-tests	`38.33% <ø> (-0.05%)`	⬇️	Carriedforward from d5dd02e
integration-tests	`34.45% <ø> (+0.62%)`	⬆️
kind-e2e-tests	`41.03% <50.00%> (-6.35%)`	⬇️
unit-tests	`59.92% <56.66%> (-2.79%)`	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
pkg/controller/egress/controller.go	`87.50% <50.00%> (-1.20%)`	⬇️
pkg/agent/controller/egress/egress_controller.go	`83.68% <59.09%> (+0.03%)`	⬆️

... and 106 files with indirect coverage changes

tnqn · 2023-04-23T09:31:53Z

/test-all

XinShuYang · 2023-04-23T14:37:43Z

/test-all

tnqn · 2023-04-28T14:27:47Z

/test-all

jianjuns · 2023-04-28T16:14:39Z

pkg/apis/crd/v1alpha2/types.go

 	// ExternalIPPool specifies the IP Pool that the EgressIP should be allocated from.
 	// If it is empty, the specified EgressIP must be assigned to a Node manually.
 	// If it is non-empty, the EgressIP will be assigned to a Node specified by the pool automatically and will failover
 	// to a different Node when the Node becomes unreachable.
-	ExternalIPPool string `json:"externalIPPool"`
+	ExternalIPPool string `json:"externalIPPool,omitempty"`


Maybe we can deprecate this field in a future release?

Do you mean both EgressIP and ExternalIPPool, the singular fields?

jianjuns · 2023-04-28T16:15:31Z

pkg/apis/crd/v1alpha2/types.go

@@ -213,11 +216,18 @@ type EgressSpec struct {
 	// If ExternalIPPool is non-empty, it can be empty and will be assigned by Antrea automatically.
 	// If both ExternalIPPool and EgressIP are non-empty, the IP must be in the pool.
 	EgressIP string `json:"egressIP,omitempty"`
+	// EgressIPs specifies multiple SNAT IP addresses for the selected workloads.


Do we require each IP must be from different ExternalIPPools?

We probably should check the uniqueness of the pools because:

Using same ExternalIPPools multiple times is meaningless

It may cause trouble to the implementation, e.g. we might use map to track pool -> ip allocation.

Ok. Would you add comments for that?

updated the comment of ExternalIPPools

Should we say IPs must be in ExternalIPPools? And move "Entries with the same index in EgressIPs and ExternalIPPools are correlated." here?

But we could also support IPs not allocated from pools. For example, users could get HA support when they have two Nodes that have public access and just use two Node IPs as the Egress IPs here.

@jianjuns If there are more comments about the fields' comment, we could update with a following up PR or when we officially support multiple IPs. I will merge it first to proceed.

Nodes in a cluster could reside in multiple subnets, but the egress IP needs to be routable in the underlay network and usually resides in the same subnet of the Node hosting it. Therefore, there may be a situation where no available nodes are eligible to host an egress IP address if all Nodes in a subnet are down, interrupting egress traffic of the workloads that use the IP. As the first step of supporting the above scenario, the patch extends the Egress IP to support multiple Egress IPs and pools so that one Egress IP can be configured for each subnet, making Egress failover across the whole cluster possible. Besides, it also adds a field, `status.egressIP`, to represent the effective Egress IP. When there is no eligible Node for any of the Egress IPs, the field will be empty. Signed-off-by: Quan Tian <qtian@vmware.com>

jianjuns · 2023-05-04T04:13:40Z

pkg/apis/crd/v1alpha2/types.go

@@ -213,11 +216,18 @@ type EgressSpec struct {
 	// If ExternalIPPool is non-empty, it can be empty and will be assigned by Antrea automatically.
 	// If both ExternalIPPool and EgressIP are non-empty, the IP must be in the pool.
 	EgressIP string `json:"egressIP,omitempty"`
+	// EgressIPs specifies multiple SNAT IP addresses for the selected workloads.


Should we say IPs must be in ExternalIPPools? And move "Entries with the same index in EgressIPs and ExternalIPPools are correlated." here?

tnqn · 2023-05-04T09:24:45Z

/skip-all

…#4603) Nodes in a cluster could reside in multiple subnets, but the egress IP needs to be routable in the underlay network and usually resides in the same subnet of the Node hosting it. Therefore, there may be a situation where no available nodes are eligible to host an egress IP address if all Nodes in a subnet are down, interrupting egress traffic of the workloads that use the IP. As the first step of supporting the above scenario, the patch extends the Egress IP to support multiple Egress IPs and pools so that one Egress IP can be configured for each subnet, making Egress failover across the whole cluster possible. Besides, it also adds a field, `status.egressIP`, to represent the effective Egress IP. When there is no eligible Node for any of the Egress IPs, the field will be empty. Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn force-pushed the multi-egress-ips branch 4 times, most recently from 62ccdf8 to 4f70047 Compare February 9, 2023 09:09

tnqn force-pushed the multi-egress-ips branch 2 times, most recently from 7e3fedc to 5ee642c Compare February 16, 2023 13:10

tnqn force-pushed the multi-egress-ips branch 2 times, most recently from 5e847e1 to 567e2b3 Compare February 24, 2023 08:09

tnqn force-pushed the multi-egress-ips branch from 567e2b3 to 58e97ac Compare April 23, 2023 07:05

tnqn changed the title ~~[WIP] Extend Egress API to support multiple Egress IPs~~ Extend Egress data structure to support multiple Egress IPs Apr 23, 2023

tnqn marked this pull request as ready for review April 23, 2023 11:35

tnqn closed this Apr 26, 2023

tnqn reopened this Apr 28, 2023

tnqn force-pushed the multi-egress-ips branch from 58e97ac to 1ab5e6d Compare April 28, 2023 14:13

tnqn requested review from antoninbas and jianjuns April 28, 2023 14:25

tnqn changed the title ~~Extend Egress data structure to support multiple Egress IPs~~ Update Egress API to support multiple Egress IPs and pools Apr 28, 2023

jianjuns previously approved these changes Apr 28, 2023

View reviewed changes

antoninbas previously approved these changes Apr 28, 2023

View reviewed changes

tnqn dismissed stale reviews from antoninbas and jianjuns via 1a8d0fe May 4, 2023 03:11

tnqn force-pushed the multi-egress-ips branch from 1ab5e6d to 1a8d0fe Compare May 4, 2023 03:11

jianjuns approved these changes May 4, 2023

View reviewed changes

tnqn merged commit e635aef into antrea-io:main May 4, 2023

tnqn deleted the multi-egress-ips branch May 4, 2023 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Egress API to support multiple Egress IPs and pools #4603

Update Egress API to support multiple Egress IPs and pools #4603

tnqn commented Feb 6, 2023 •

edited

Loading

codecov bot commented Feb 6, 2023 •

edited

Loading

tnqn commented Apr 23, 2023

XinShuYang commented Apr 23, 2023

tnqn commented Apr 28, 2023

jianjuns Apr 28, 2023

tnqn May 1, 2023

jianjuns May 1, 2023

jianjuns Apr 28, 2023

tnqn May 1, 2023

jianjuns May 1, 2023

tnqn May 4, 2023

jianjuns May 4, 2023

tnqn May 4, 2023

tnqn May 4, 2023

jianjuns May 4, 2023

tnqn commented May 4, 2023

Update Egress API to support multiple Egress IPs and pools #4603

Update Egress API to support multiple Egress IPs and pools #4603

Conversation

tnqn commented Feb 6, 2023 • edited Loading

codecov bot commented Feb 6, 2023 • edited Loading

Codecov Report

tnqn commented Apr 23, 2023

XinShuYang commented Apr 23, 2023

tnqn commented Apr 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn commented May 4, 2023

tnqn commented Feb 6, 2023 •

edited

Loading

codecov bot commented Feb 6, 2023 •

edited

Loading