-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
better error information for vault client error #5139
Comments
Thanks @camerondavison! I agree that providing some context here in the error message here would be great. I will create a PR for it soon unless you beat me to it ;-). I generally like to wrap errors in the format of |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Took me a while to track down the actual problem so I thought that it would be nice to have some better bread crumbs along the way if possible.
The vault token on our nomad server expired because the ssl cert on the vault server expired and we did not catch it until the TTL of the vault token had already expired. In our nomad logs I found.
failed to renew Vault token before lease expiration
which ultimately led me to the problem and solution of getting new vault tokens for nomad.The journey started with an error that I saw in the allocation.
vault: server error deriving vault token: Put https://vault.service.consul:8200/v1/auth/token/renew-self: x509: certificate has expired or is not yet valid
Which is the error that was given back by the server.
[ERR] nomad.node: Vault token creation for alloc "12e128f9-0808-adb7-3a3a-3ce5adffd0df" failed: Put https://vault.service.consul:8200/v1/auth/token/renew-self: x509: certificate has expired or is not yet valid
Which I am assuming was the string on
connEstablishedErr
nomad/nomad/vault.go
Lines 516 to 525 in c64b41d
renew
error that happened. Which is then used bynomad/nomad/vault.go
Line 908 in c64b41d
Seems like putting something like
Vault connection is not established: Error %v
or something prepended to that error would be helpful. Or probably even better just prependnomad/nomad/vault.go
Line 523 in c64b41d
Failed to renew vault token before expiration because %v
.Nomad version
Nomad v0.8.6 (ab54ebc+CHANGES)
Operating system and Environment details
linux
The text was updated successfully, but these errors were encountered: