Document Egress feature's options and usage on cloud #5436

tnqn · 2023-08-24T15:47:06Z

The commit covers AWS only, other clouds will be added after more experiments.

antoninbas · 2023-08-24T18:06:50Z

docs/egress.md

+  be SNAT'd by Egresses. The option was added in Antrea v1.4.0.
+- `egress.maxEgressIPsPerNode` - The maximum number of Egress IPs that can be
+  assigned to a Node. It's useful when the Node network restricts the number of
+  secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The


Suggested change

secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The

secondary IPs a Node can have, e.g. in AWS EC2. The configured value must not be greater than 255. The

antoninbas · 2023-08-24T18:07:28Z

docs/egress.md

+  secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The
+  restriction applies to all Nodes in the cluster. If you want to set different
+  capacities for Nodes, the `node.antrea.io/max-egress-ips` annotation of Node
+  objects can be used to specify distinct values for Nodes, taking priority


Suggested change

objects can be used to specify distinct values for Nodes, taking priority

objects can be used to specify different values for different Nodes, taking priority

antoninbas · 2023-08-24T18:14:42Z

docs/egress.md

+In Amazon VPC, every packet flow on the network is individually authorized
+to validate the source IP and destination IP before it is transmitted and


I think it is more than that, as this could be worked around by disabling address checks. Isn't the core problem that ARP traffic doesn't hit the network? Maybe it would be useful to link to https://docs.aws.amazon.com/whitepapers/latest/logical-separation/vpc-and-accompanying-features.html:

Every packet flow on the network is individually authorized against a rule to validate the correct source and destination before it is transmitted and delivered. It is highly improbable for information to arbitrarily pass between entities without specifically being authorized by both the transmitting and receiving entity. If a packet is being routed to a destination without a rule that matches it, the packet is dropped. Reply addresses must be valid or the packet is dropped. Moreover, while address resolution protocol (ARP) packets trigger an authenticated database look-up, ARP packets never hit the network as they are not needed for discovery of the virtual network topology. This means ARP spoofing is highly improbable on the AWS network.

Yes, updated.

antoninbas · 2023-08-24T18:17:45Z

docs/egress.md

+of advertising the Egress IPs to the Node network via ARP or NDP protocols.
+However, cloud networks usually apply SpoofGuard which prevents the Nodes from
+using any IP that is not configured for them in the cloud's control plane, or
+even don't support multicast and broadcast. These lead to the high availability


These restrictions lead to High-Availability Egress not being as readily available on some clouds as it is on on-premise networks, and some custom (i.e., cloud-specific) work is required to assign the Egress IPs as secondary IPs of the Nodes in the cloud's control plane.

github-actions · 2023-11-24T00:03:36Z

This PR is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

antoninbas · 2023-11-27T18:47:27Z

@tnqn what happened to this PR?

tnqn · 2023-11-28T16:20:01Z

@tnqn what happened to this PR?

Thanks for reminding, will update soon.

antoninbas

some more nits and suggestions

antoninbas · 2023-12-06T17:16:01Z

docs/egress.md

@@ -333,6 +336,61 @@ another Node's IP can recover the egress connection. Antrea will detect the
 configuration change and redirect the packets from the Pods in the `prod`
 Namespace to the new Node.

+## Options


nit: s/Options/Configuration options

antoninbas · 2023-12-06T17:18:43Z

docs/egress.md

+
+High-Availability Egress requires the Egress IPs to be able to float across
+Nodes. When assigning an Egress IP to a Node, Antrea assumes the responsibility
+of advertising the Egress IPs to the Node network via ARP or NDP protocols.


via the ARP or NDP protocols.

antoninbas · 2023-12-06T17:19:55Z

docs/egress.md

+on on-premise networks, and some custom (i.e., cloud-specific) work is required
+to assign the Egress IPs as secondary IPs of the Nodes in the cloud's control
+plane.


maybe this would be clearer:

and some custom (i.e., cloud-specific) work is required in the cloud's control plane to assign the Egress IP as secondary Node IPs.

antoninbas · 2023-12-06T17:23:09Z

docs/egress.md

+### AWS
+
+In Amazon VPC, ARP packets never hit the network, and traffic with Egress IP as
+source IP or destination IP isn't transmitted arbitrarily before they are


do you mean "unless they are explicitly authorized" instead of "before they are authorized"?

yes, updated

antoninbas · 2023-12-06T17:25:08Z

docs/egress.md

+If you are using High-Availability Egress and let Antrea manage the Egress IPs'
+assignment, at the moment Antrea can only assign the Egress IP to an Egress


If you are using High-Availability Egress and let Antrea manage the assignment of Egress IPs: at the moment Antrea can only assign the Egress IP to an Egress Node at the operating system level (i.e., add the IP to the interface), and you still need to ensure the Egress IP ...

antoninbas · 2023-12-06T17:26:53Z

docs/egress.md

+If you are using static Egress and managing the Egress IPs' assignment
+yourself, you should ensure the Egress IP is assigned as one of the IP


If you are using static Egress and managing the assignment of Egress IPs yourself: you should ensure...

The commit covers AWS only, other clouds will be added after more experiments. Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn

comments addressed, thanks @antoninbas

tnqn · 2023-12-07T03:20:37Z

docs/egress.md

@@ -333,6 +336,61 @@ another Node's IP can recover the egress connection. Antrea will detect the
 configuration change and redirect the packets from the Pods in the `prod`
 Namespace to the new Node.

+## Options


tnqn · 2023-12-07T03:20:42Z

docs/egress.md

+
+High-Availability Egress requires the Egress IPs to be able to float across
+Nodes. When assigning an Egress IP to a Node, Antrea assumes the responsibility
+of advertising the Egress IPs to the Node network via ARP or NDP protocols.


tnqn · 2023-12-07T03:21:17Z

docs/egress.md

+on on-premise networks, and some custom (i.e., cloud-specific) work is required
+to assign the Egress IPs as secondary IPs of the Nodes in the cloud's control
+plane.


tnqn · 2023-12-07T03:22:53Z

docs/egress.md

+### AWS
+
+In Amazon VPC, ARP packets never hit the network, and traffic with Egress IP as
+source IP or destination IP isn't transmitted arbitrarily before they are


yes, updated

tnqn · 2023-12-07T03:24:20Z

docs/egress.md

+If you are using static Egress and managing the Egress IPs' assignment
+yourself, you should ensure the Egress IP is assigned as one of the IP


tnqn · 2023-12-07T03:35:22Z

docs/egress.md

+If you are using High-Availability Egress and let Antrea manage the Egress IPs'
+assignment, at the moment Antrea can only assign the Egress IP to an Egress


antoninbas

LGTM

tnqn · 2023-12-07T03:46:49Z

/skip-all

tnqn added the kind/documentation Categorizes issue or PR as related to a documentation. label Aug 24, 2023

tnqn requested review from antoninbas and jianjuns August 24, 2023 15:47

antoninbas reviewed Aug 24, 2023

View reviewed changes

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 24, 2023

github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 28, 2023

tnqn force-pushed the egress-doc branch from 22f2e1c to ad20618 Compare December 6, 2023 11:24

tnqn added the area/transit/egress Issues or PRs related to Egress (SNAT for traffic egressing the cluster). label Dec 6, 2023

antoninbas reviewed Dec 6, 2023

View reviewed changes

Document Egress feature's options and usage on cloud

7f1b7a2

The commit covers AWS only, other clouds will be added after more experiments. Signed-off-by: Quan Tian <qtian@vmware.com>

tnqn force-pushed the egress-doc branch from ad20618 to 7f1b7a2 Compare December 7, 2023 03:38

tnqn commented Dec 7, 2023

View reviewed changes

antoninbas approved these changes Dec 7, 2023

View reviewed changes

tnqn merged commit 7c45c22 into antrea-io:main Dec 7, 2023
47 of 50 checks passed

tnqn deleted the egress-doc branch December 7, 2023 03:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document Egress feature's options and usage on cloud #5436

Document Egress feature's options and usage on cloud #5436

tnqn commented Aug 24, 2023

antoninbas Aug 24, 2023

tnqn Aug 25, 2023

antoninbas Aug 24, 2023

tnqn Dec 6, 2023

antoninbas Aug 24, 2023

tnqn Dec 6, 2023

antoninbas Aug 24, 2023

tnqn Dec 6, 2023

github-actions bot commented Nov 24, 2023

antoninbas commented Nov 27, 2023

tnqn commented Nov 28, 2023

antoninbas left a comment

antoninbas Dec 6, 2023

tnqn Dec 7, 2023

antoninbas Dec 6, 2023

tnqn Dec 7, 2023

antoninbas Dec 6, 2023

tnqn Dec 7, 2023

antoninbas Dec 6, 2023

tnqn Dec 7, 2023

antoninbas Dec 6, 2023

tnqn Dec 7, 2023

antoninbas Dec 6, 2023

tnqn Dec 7, 2023

tnqn left a comment

tnqn Dec 7, 2023

tnqn Dec 7, 2023

tnqn Dec 7, 2023

tnqn Dec 7, 2023

tnqn Dec 7, 2023

tnqn Dec 7, 2023

antoninbas left a comment

tnqn commented Dec 7, 2023

	secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The
	secondary IPs a Node can have, e.g. in AWS EC2. The configured value must not be greater than 255. The

	objects can be used to specify distinct values for Nodes, taking priority
	objects can be used to specify different values for different Nodes, taking priority

		In Amazon VPC, every packet flow on the network is individually authorized
		to validate the source IP and destination IP before it is transmitted and

		If you are using High-Availability Egress and let Antrea manage the Egress IPs'
		assignment, at the moment Antrea can only assign the Egress IP to an Egress

		If you are using static Egress and managing the Egress IPs' assignment
		yourself, you should ensure the Egress IP is assigned as one of the IP

Document Egress feature's options and usage on cloud #5436

Document Egress feature's options and usage on cloud #5436

Conversation

tnqn commented Aug 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 24, 2023

antoninbas commented Nov 27, 2023

tnqn commented Nov 28, 2023

antoninbas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoninbas left a comment

Choose a reason for hiding this comment

tnqn commented Dec 7, 2023