Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpc: use tls wrapped connection for streaming rpc #5954

Merged
merged 2 commits into from
Jul 12, 2019
Merged

rpc: use tls wrapped connection for streaming rpc #5954

merged 2 commits into from
Jul 12, 2019

Commits on Jul 12, 2019

  1. rpc: add positive tests for server streaming RPC

    Mahmood Ali committed Jul 12, 2019
    Configuration menu
    Copy the full SHA
    a6604f8 View commit details
    Browse the repository at this point in the history

  2. rpc: use tls wrapped connection for streaming rpc 

    This ensures that server-to-server streaming RPC calls use the tls
wrapped connections.

Prior to this, `streamingRpcImpl` function uses tls for setting header
and invoking the rpc method, but returns unwrapped tls connection.
Thus, streaming writes fail with tls errors.

This tls streaming bug existed since 0.8.0[1], but PR #5654[2]
exacerbated it in 0.9.2.  Prior to PR #5654, nomad client used to
shuffle servers at every heartbeat -- `servers.Manager.setServers`[3]
always shuffled servers and was called by heartbeat code[4].  Shuffling
servers meant that a nomad client would heartbeat and establish a
connection against all nomad servers eventually.  When handling
streaming RPC calls, nomad servers used these local connection to
communicate directly to the client.  The server-to-server forwarding
logic was left mostly unexercised.

PR #5654 means that a nomad client may connect to a single server only
and caused the server-to-server forward streaming RPC code to get
exercised more and unearthed the problem.

[1] https://github.com/hashicorp/nomad/blob/v0.8.0/nomad/rpc.go#L501-L515
[2] #5654
[3] https://github.com/hashicorp/nomad/blob/v0.9.1/client/servers/manager.go#L198-L216
[4] https://github.com/hashicorp/nomad/blob/v0.9.1/client/client.go#L1603
    Mahmood Ali committed Jul 12, 2019
    Configuration menu
    Copy the full SHA
    b0d98d1 View commit details
    Browse the repository at this point in the history