-
Notifications
You must be signed in to change notification settings - Fork 359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Document Egress feature's options and usage on cloud #5436
Conversation
docs/egress.md
Outdated
be SNAT'd by Egresses. The option was added in Antrea v1.4.0. | ||
- `egress.maxEgressIPsPerNode` - The maximum number of Egress IPs that can be | ||
assigned to a Node. It's useful when the Node network restricts the number of | ||
secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The | |
secondary IPs a Node can have, e.g. in AWS EC2. The configured value must not be greater than 255. The |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
secondary IPs a Node can have, e.g. EKS. It must not be greater than 255. The | ||
restriction applies to all Nodes in the cluster. If you want to set different | ||
capacities for Nodes, the `node.antrea.io/max-egress-ips` annotation of Node | ||
objects can be used to specify distinct values for Nodes, taking priority |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
objects can be used to specify distinct values for Nodes, taking priority | |
objects can be used to specify different values for different Nodes, taking priority |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
In Amazon VPC, every packet flow on the network is individually authorized | ||
to validate the source IP and destination IP before it is transmitted and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is more than that, as this could be worked around by disabling address checks. Isn't the core problem that ARP traffic doesn't hit the network? Maybe it would be useful to link to https://docs.aws.amazon.com/whitepapers/latest/logical-separation/vpc-and-accompanying-features.html:
Every packet flow on the network is individually authorized against a rule to validate the correct source and destination before it is transmitted and delivered. It is highly improbable for information to arbitrarily pass between entities without specifically being authorized by both the transmitting and receiving entity. If a packet is being routed to a destination without a rule that matches it, the packet is dropped. Reply addresses must be valid or the packet is dropped. Moreover, while address resolution protocol (ARP) packets trigger an authenticated database look-up, ARP packets never hit the network as they are not needed for discovery of the virtual network topology. This means ARP spoofing is highly improbable on the AWS network.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, updated.
docs/egress.md
Outdated
of advertising the Egress IPs to the Node network via ARP or NDP protocols. | ||
However, cloud networks usually apply SpoofGuard which prevents the Nodes from | ||
using any IP that is not configured for them in the cloud's control plane, or | ||
even don't support multicast and broadcast. These lead to the high availability |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These restrictions lead to High-Availability Egress not being as readily available on some clouds as it is on on-premise networks, and some custom (i.e., cloud-specific) work is required to assign the Egress IPs as secondary IPs of the Nodes in the cloud's control plane.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
This PR is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days |
@tnqn what happened to this PR? |
Thanks for reminding, will update soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some more nits and suggestions
docs/egress.md
Outdated
@@ -333,6 +336,61 @@ another Node's IP can recover the egress connection. Antrea will detect the | |||
configuration change and redirect the packets from the Pods in the `prod` | |||
Namespace to the new Node. | |||
|
|||
## Options |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: s/Options/Configuration options
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
|
||
High-Availability Egress requires the Egress IPs to be able to float across | ||
Nodes. When assigning an Egress IP to a Node, Antrea assumes the responsibility | ||
of advertising the Egress IPs to the Node network via ARP or NDP protocols. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
via the ARP or NDP protocols.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
on on-premise networks, and some custom (i.e., cloud-specific) work is required | ||
to assign the Egress IPs as secondary IPs of the Nodes in the cloud's control | ||
plane. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe this would be clearer:
and some custom (i.e., cloud-specific) work is required in the cloud's control plane to assign the Egress IP as secondary Node IPs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
### AWS | ||
|
||
In Amazon VPC, ARP packets never hit the network, and traffic with Egress IP as | ||
source IP or destination IP isn't transmitted arbitrarily before they are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you mean "unless they are explicitly authorized" instead of "before they are authorized"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, updated
docs/egress.md
Outdated
If you are using High-Availability Egress and let Antrea manage the Egress IPs' | ||
assignment, at the moment Antrea can only assign the Egress IP to an Egress |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you are using High-Availability Egress and let Antrea manage the assignment of Egress IPs: at the moment Antrea can only assign the Egress IP to an Egress Node at the operating system level (i.e., add the IP to the interface), and you still need to ensure the Egress IP ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
If you are using static Egress and managing the Egress IPs' assignment | ||
yourself, you should ensure the Egress IP is assigned as one of the IP |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you are using static Egress and managing the assignment of Egress IPs yourself: you should ensure...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
The commit covers AWS only, other clouds will be added after more experiments. Signed-off-by: Quan Tian <qtian@vmware.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
comments addressed, thanks @antoninbas
docs/egress.md
Outdated
@@ -333,6 +336,61 @@ another Node's IP can recover the egress connection. Antrea will detect the | |||
configuration change and redirect the packets from the Pods in the `prod` | |||
Namespace to the new Node. | |||
|
|||
## Options |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
|
||
High-Availability Egress requires the Egress IPs to be able to float across | ||
Nodes. When assigning an Egress IP to a Node, Antrea assumes the responsibility | ||
of advertising the Egress IPs to the Node network via ARP or NDP protocols. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
on on-premise networks, and some custom (i.e., cloud-specific) work is required | ||
to assign the Egress IPs as secondary IPs of the Nodes in the cloud's control | ||
plane. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
### AWS | ||
|
||
In Amazon VPC, ARP packets never hit the network, and traffic with Egress IP as | ||
source IP or destination IP isn't transmitted arbitrarily before they are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, updated
docs/egress.md
Outdated
If you are using static Egress and managing the Egress IPs' assignment | ||
yourself, you should ensure the Egress IP is assigned as one of the IP |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
docs/egress.md
Outdated
If you are using High-Availability Egress and let Antrea manage the Egress IPs' | ||
assignment, at the moment Antrea can only assign the Egress IP to an Egress |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/skip-all |
The commit covers AWS only, other clouds will be added after more experiments.