Nomad client RPC communication errors after cert rotation + SIGHUP #4413
Comments
SoMuchToGrok
changed the title
SoMuchToGrok
changed the title
|
@SoMuchToGrok Can you provide sample certificates that demonstrate this behavior? Also are you seeing the EOF forever after sending the signal or just immediately after. We do close connections after reloading the certificates so this would be expected behavior. Sustained errors would be bad though. |
|
I'm seeing the EOF forever (well...at least 24+ hours and counting; going to assume it will keep going on). I'm additionally seeing secrets expire on the Vault side, but nomad is continuing to use them (presumably it'll try to keep using the secret even after it fails to communicate with Vault). I'll work on generating some sample certificates, including the full chain, and will update when I have them. |
|
Just chiming in: I believe we've encountered this issue on a Nomad 0.8.4/Ubuntu 16.04 cluster. After updating certificates (issued via Let's Encrypt) and issuing a SIGHUP to reload the Nomad server instances, clients began reporting I'm not yet sure if it's relevant, but our tooling issued the SIGHUP against all three Nomad servers within about a second of one another. We will try to establish a reproduction case and report back. /edit restarting the Nomad server instances resolved the issue. |
|
As a reproduction test case, I created and distributed self-signed certificates for every node (3x servers, 2x clients in our case) and issued a SIGHUP against all Nomad services. After repeatedly swapping between Let's Encrypt and self-signed key material then triggering a reload, I've been unable to reproduce the scenario above. I'll keep trying. |
|
Helo, I'm working with @SoMuchToGrok and attached should be a full chain of certificates: RootNomad ServerNomad ClientPlease let us know if there's any more information we can provide to help you with this. |
|
Is there anything additional we can do to help with this issue? We're still stuck on Nomad v0.7.1 because of this bug. |
|
I was able to reproduce with the following:
When you re-issue a cert and send a SIGHUP to the nomad client, everything is fine. It's when you re-issue a cert for the nomad server that the problem manifests itself. Nomad Server LogsYou can see after the SIGHUP is sent that there are a lot of errors like nomad.rpc: Closing server RPC connection Nomad Client LogsShortly after the nomad server starts erroring, you can see rpc errors happen in the client over and over. Nomad CLI LogsWhile this chaos is going down, if you try to do a Although if I let the server / client run for a few more minutes, the CLI will return data but I will be unable to place new jobs (which is expected I think) |
Xopherus
added a commit
to Xopherus/nomad
that referenced
this issue
Aug 10, 2018
Multiplexer continues to create rpc connections even when the context which is passed to the underlying rpc connections is cancelled by the server. This was causing hashicorp#4413 - when a SIGHUP causes everything to reload, it uses context to cancel the underlying http/rpc connections so that they may come up with the new configuration. The multiplexer was not being cancelled properly so it would continue to create rpc connections and constantly fail, causing communication issues with other nomad agents. Fixes hashicorp#4413
|
I believe I found the issue! I will open a PR with a proposed fix - let me know if that will be acceptable! |
Xopherus
added a commit
to Xopherus/nomad
that referenced
this issue
Aug 10, 2018
Multiplexer continues to create rpc connections even when the context which is passed to the underlying rpc connections is cancelled by the server. This was causing hashicorp#4413 - when a SIGHUP causes everything to reload, it uses context to cancel the underlying http/rpc connections so that they may come up with the new configuration. The multiplexer was not being cancelled properly so it would continue to create rpc connections and constantly fail, causing communication issues with other nomad agents. Fixes hashicorp#4413
Xopherus
added a commit
to Xopherus/nomad
that referenced
this issue
Aug 13, 2018
Multiplexer continues to create rpc connections even when the context which is passed to the underlying rpc connections is cancelled by the server. This was causing hashicorp#4413 - when a SIGHUP causes everything to reload, it uses context to cancel the underlying http/rpc connections so that they may come up with the new configuration. The multiplexer was not being cancelled properly so it would continue to create rpc connections and constantly fail, causing communication issues with other nomad agents. Fixes hashicorp#4413
Xopherus
added a commit
to Xopherus/nomad
that referenced
this issue
Aug 13, 2018
Multiplexer continues to create rpc connections even when the context which is passed to the underlying rpc connections is cancelled by the server. This was causing hashicorp#4413 - when a SIGHUP causes everything to reload, it uses context to cancel the underlying http/rpc connections so that they may come up with the new configuration. The multiplexer was not being cancelled properly so it would continue to create rpc connections and constantly fail, causing communication issues with other nomad agents. Fixes hashicorp#4413
preetapan
pushed a commit
that referenced
this issue
Sep 10, 2018
Multiplexer continues to create rpc connections even when the context which is passed to the underlying rpc connections is cancelled by the server. This was causing #4413 - when a SIGHUP causes everything to reload, it uses context to cancel the underlying http/rpc connections so that they may come up with the new configuration. The multiplexer was not being cancelled properly so it would continue to create rpc connections and constantly fail, causing communication issues with other nomad agents. Fixes #4413
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Nomad version
Nomad v0.8.3 (c85483d)
Operating system and Environment details
Ubuntu 16.04
Consul v1.0.6
Vault v0.10.2
Issue
Since upgrading from v0.7.1 to v0.8.3, nomad client -> server RPC communication appears to error out continuously after I rotate my TLS certificates and send a SIGHUP to the nomad client PID. Additionally, all jobs with Vault integrations that are scheduled after the SIGHUP has been sent fail with similar RPC EOF errors. I have been able to reproduce this in two different environments.
Reproduction steps
Deploy a cluster of nomad servers and nomad clients (>1 each).
Generate TLS certificates and enable TLS for all servers and clients
Generate a new certificate for a nomad client (signed by the same CA as the previously used certificate) and overwrite cert_file and key_file with the new contents (don't append to the existing files, overwrite them completely). The previously used certificate should still be valid (not expired).
Send the nomad client process a SIGHUP
Validate that nomad client logs are throwing RPC errors when communicating to the nomad servers.
Schedule a job that uses the Vault integration, and validate that tasks are unable to create vault tokens due to similar (perhaps same) RPC EOF errors.
Nomad Server logs
Could not find anything relevant. It appears that this issue is not present in the servers, as I have reproduced the same steps without any known issues.
Nomad Client logs
Task state logs (from vault-integrated jobs scheduled after the SIGHUP has been sent)
The text was updated successfully, but these errors were encountered: