better error information for vault client error

[camerondavison]

Took me a while to track down the actual problem so I thought that it would be nice to have some better bread crumbs along the way if possible.

The vault token on our nomad server expired because the ssl cert on the vault server expired and we did not catch it until the TTL of the vault token had already expired. In our nomad logs I found.
failed to renew Vault token before lease expiration which ultimately led me to the problem and solution of getting new vault tokens for nomad.

The journey started with an error that I saw in the allocation.
vault: server error deriving vault token: Put https://vault.service.consul:8200/v1/auth/token/renew-self: x509: certificate has expired or is not yet valid

Which is the error that was given back by the server.
[ERR] nomad.node: Vault token creation for alloc "12e128f9-0808-adb7-3a3a-3ce5adffd0df" failed: Put https://vault.service.consul:8200/v1/auth/token/renew-self: x509: certificate has expired or is not yet valid

Which I am assuming was the string on connEstablishedErr

nomad/nomad/vault.go

Lines 516 to 525 in c64b41d

	backoff = nextBackoff(backoff, currentExpiration)
	if backoff < 0 {
	// We have failed to renew the token past its expiration. Stop
	// renewing with Vault.
	v.logger.Error("failed to renew Vault token before lease expiration. Shutting down Vault client")
	v.l.Lock()
	v.connEstablished = false
	v.connEstablishedErr = err
	v.l.Unlock()
	return

which was taken from the renew error that happened. Which is then used by

nomad/nomad/vault.go

Line 908 in c64b41d

return nil, err

Seems like putting something like Vault connection is not established: Error %v or something prepended to that error would be helpful. Or probably even better just prepend

nomad/nomad/vault.go

Line 523 in c64b41d

v.connEstablishedErr = err

with Failed to renew vault token before expiration because %v.

Nomad version

Nomad v0.8.6 (ab54ebc+CHANGES)

Operating system and Environment details

linux

[notnoop]

Thanks @camerondavison! I agree that providing some context here in the error message here would be great. I will create a PR for it soon unless you beat me to it ;-).

I generally like to wrap errors in the format of failed to establish Vault connection: %v.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.