server: node connections must not be forwarded #7370
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This fixes a bug where a forwarded node update request may be assumed to be the actual direct client connection if the server just lost leadership. When a nomad non-leader server receives a Node.UpdateStatus request, it forwards the RPC request to the leader, and holds on the request Yamux connection in a cache to allow for server<->client forwarding. When the leader handles the request, it must differentiate between a forwarded connection vs the actual connection. This is done in https://github.com/hashicorp/nomad/blob/v0.10.4/nomad/node_endpoint.go#L412 Now, consider if the non-leader server forwards to the connection to a recently deposed nomad leader, which in turn forwards the RPC request to the new leader. Without this change, the deposed leader will mistake the forwarded connection for the actual client connection and cache it mapped to the client ID. If the server attempts to connect to that client, it will attempt to start a connection/session to the other server instead and the call will hang forever. This change ensures that we only add node connection mapping if the request is not a forwarded request, regardless of circumstances.
schmichael
approved these changes
Mar 17, 2020
There was a problem hiding this comment.
Great catch! It looks like we actually handled this properly in 3/6 cases which makes me want to pass the bool to addNodeConn to avoid this programming error in the future. Not required, but it seems like an easy problem to make and a hard problem to test for.
notnoop
force-pushed
the
b-leader-forwarding-client-detection
branch
from
e71160a
to
18173fa
Compare
notnoop
force-pushed
the
b-leader-forwarding-client-detection
branch
from
18173fa
to
63b8e45
Compare
notnoop
pushed a commit
that referenced
this pull request
Apr 9, 2020
…tection server: node connections must not be forwarded
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes a bug where a forwarded node update request may be assumed
to be the actual direct client connection if the server just lost
leadership.
When a nomad non-leader server receives a Node.UpdateStatus request, it
forwards the RPC request to the leader, and holds on the request
Yamux connection in a cache to allow for server<->client forwarding.
When the leader handles the request, it must differentiate between a
forwarded connection vs the actual connection. This is done in
https://github.com/hashicorp/nomad/blob/v0.10.4/nomad/node_endpoint.go#L412
Now, consider if the non-leader server forwards to the connection to a
recently deposed nomad leader, which in turn forwards the RPC request to
the new leader.
Without this change, the deposed leader will mistake the forwarded
connection for the actual client connection and cache it mapped to the
client ID. If the server attempts to connect to that client, it will
attempt to start a connection/session to the other server instead and
the call will hang forever.
This change ensures that we only add node connection mapping if the
request is not a forwarded request, regardless of circumstances.
Fixes #4604