-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SIGSEGV when seems vault have stale state #4921
Comments
Seems that code must be changed like this: diff --git a/nomad/vault.go b/nomad/vault.go
index fc60075fb..05829b3d4 100644
--- a/nomad/vault.go
+++ b/nomad/vault.go
@@ -595,8 +595,9 @@ func (v *vaultClient) parseSelfToken() error {
if err != nil {
return fmt.Errorf("failed to lookup Vault periodic token: %v", err)
}
+ } else {
+ self = secret
}
- self = secret
// Read and parse the fields
var data tokenData Because absolutely assignment |
or may be changed a little bit simpler, like this: diff --git a/nomad/vault.go b/nomad/vault.go
index fc60075fb..ca59bf356 100644
--- a/nomad/vault.go
+++ b/nomad/vault.go
@@ -588,7 +588,7 @@ func (v *vaultClient) parseSelfToken() error {
var self *vapi.Secret
// Try looking up the token using the self endpoint
- secret, err := auth.LookupSelf()
+ self, err := auth.LookupSelf()
if err != nil {
// Try looking up our token directly
self, err = auth.Lookup(v.client.Token())
@@ -596,7 +596,6 @@ func (v *vaultClient) parseSelfToken() error {
return fmt.Errorf("failed to lookup Vault periodic token: %v", err)
}
}
- self = secret
// Read and parse the fields
var data tokenData |
@preetapan what do you think about this? |
@tantra35 thanks for reporting this. This looks similar to another sigsegv we fixed recently in https://github.com/hashicorp/nomad/pull/4904/files. The root cause is that the vendored Vault client library returns nil(
We are planning a 0.8.7 release so we'll fix this and will be auditing the vault code again for similar issues. |
vault: protect against empty Vault secret response Fixes #4921 Sadly, we don't have proper mechanism to mock Vault client, so not sure how to best test this. I inspected the Vault client interactions, specially for cases where returned value is nil even if the error is also nil. I believe we covered all correctly now: * [`v.client.Sys().InitStatus()`](https://github.com/hashicorp/nomad/blob/f3853f11daa51336a2d46d883b1aff6feeddc7ec/nomad/vault.go#L427) - the value is non-nil boolean * [`v.client.Sys().CapabilitiesSelf(path)`](https://github.com/hashicorp/nomad/blob/f3853f11daa51336a2d46d883b1aff6feeddc7ec/nomad/vault.go#L812): Capabilities handles empty bodies in [`hasCapability`](https://github.com/hashicorp/nomad/blob/f3853f11daa51336a2d46d883b1aff6feeddc7ec/vendor/github.com/hashicorp/vault/api/sys_capabilities.go#L43-L45) - also the `nil` array is handled with proper fail-safe default. * [`v.client.Logical().Read(fmt.Sprintf("auth/token/roles/%s", role))`](https://github.com/hashicorp/nomad/blob/f3853f11daa51336a2d46d883b1aff6feeddc7ec/nomad/vault.go#L834-L840) handles when `rsecret` is nil
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.8.6
Issue
We found this in our server logs
This happens after demaged vault in our enviroment, and when we restore it state(vault) from backup
The text was updated successfully, but these errors were encountered: