-
Notifications
You must be signed in to change notification settings - Fork 9.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Supporting setting LocalAddr
in peer communication
#17068
Comments
@ahrtr |
Thanks @highpon . ping @serathius @jmhbnz @wenjiaswe @fuweid for feedback. |
cc @aojea for opinion as well |
I can see that there are multiple flags that declare addresses identifying the client, should you not use one of those since those are known to be reachable? |
Thanks for the feedback. Yes, I was planning to reuse the
Another problem is that it's hard to verify it in workflow CI. For example, how to create multiple IP addresses on the environment. |
In an etcd cluster there are some strong requirements or network connectivity between the peers, my experience is that every advanced networking knob like this can have also an undesired effect of providing a footgun for users that starts to implement complex scenarios just because they can or because they came with that idea and then they hit a problem and they start to ask for new knobs. |
Yes, agreed on this. It's a real problem discovered in our testing when etcd and kube-vip coexist. Just as I mentioned in #17068 (comment), When an etcd member accepts a connection from a peer, it checks the peer's remote address against the Subject Alternative Name (SAN) field in the peer's certificate, and reject the connection if they don't match. It's exactly the reason why the LocalAddr matters here. But when kube-vip coexist with etcd on the same host, kube-vip may dynamically add a floating IP to the host. Especially for the case of IPv6, it adds the IP to the beginning of the interface's IP list. Since etcd doesn't specify the LocalAddr when communicating with a peer, the floating IP is automatically chosen as the I believe this is also a common issue. It applies the cases where the hosts on which the etcd runs have multiple IP addresses, especially dynamical floating IP. Explicitly, it would be great if we can get feedback from network experts on:
|
that makes sense, then add just an option to define the localAddress for connecting to the apiserver, if is empty just don't set the localAddr field, this will make the change completely backwards compatible and you don't have to worry about the future.
some ideas, if you are in the same node you can play with the localhost address 127.0.0.1 and the one in the node
|
Thanks for the suggestion. |
@highpon Please follow comments below to update #17085. Thanks
|
I've tried to pick up the ball to implement these changes here: https://github.com/flawedmatrix/etcd/tree/fix/17068 One question I had was whether we only wanted to set LocalAddr to an IP address, or should we try to resolve the DNS name to an address from
After revisiting this issue, I've resolved the issue of how to e2e test this feature. |
The answer should be the latter based on #17068 (comment) |
What would you like to be added?
Currently in peer communication, the local address is automatically chosen by the golang standard lib,
etcd/client/pkg/transport/transport.go
Lines 33 to 43 in d8f0771
We should support setting a LocalAddr, which defaults to the advertise Peer address if enabled. It should be disabled by default to keep the existing behavior unless users intentionally enable it.
Why is this needed?
When accepting a connection from a peer, each etcd member checks the peer's remote address against the Subject Alternative Name (SAN) field in the peer's certificate, and reject the connection if they don't match. Please read https://etcd.io/docs/v3.5/op-guide/security/#notes-for-tls-authentication.
See also
etcd/client/pkg/transport/listener_tls.go
Lines 192 to 221 in d8f0771
Sometimes the peer may has multiple IP addresses, even including dynamic (i.e. floating) IP addresses. Probably only part of the IP addresses are valid against the SAN field in the certificate. But if an IP address which doesn't match the SAN in certificate is chosen by the golang standard lib, then it will fail to communicate with other etcd members.
The text was updated successfully, but these errors were encountered: