-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad retries indefinitely to revoke non-existant SI token(s) #17833
Comments
Hi @seanamos! I took a quick look and Nomad is assuming that this endpoint will not return an error (ref // Consul will no-op the deletion of a non-existent token (no error)
_, err := c.aclClient.TokenDelete(accessor.AccessorID, &api.WriteOptions{Namespace: accessor.ConsulNamespace})
return err Unfortunately it looks like this assumption was broken in a breaking change in Consul 1.15.0:
This isn't otherwise documented in Consul's Delete Token API docs. So it looks like an obvious cross-compatibility bug and we'll need to fix that.
Unfortunately it looks like the accessors are stored in Raft and then whenever a new leader comes on it goes thru those accessors to determine which ones can be revoked. Let me check in with some of my colleagues to see if we can figure out as workaround for you. |
In Consul 1.15.0, the Delete Token API was changed so as to return an error when deleting a non-existent ACL token. This means that if Nomad successfully deletes the token but fails to persist that fact, it will get stuck trying to delete a non-existent token forever. Update the token deletion function to ignore "not found" errors and treat them as successful deletions. Fixes: #17833
@seanamos I've got a patch up in #17847 for that. That'll get shipped in the upcoming 1.6.0 but also backported once 1.6.0 is GA (in the next couple weeks). But once that lands in |
In Consul 1.15.0, the Delete Token API was changed so as to return an error when deleting a non-existent ACL token. This means that if Nomad successfully deletes the token but fails to persist that fact, it will get stuck trying to delete a non-existent token forever. Update the token deletion function to ignore "not found" errors and treat them as successful deletions. Fixes: #17833
…17847) In Consul 1.15.0, the Delete Token API was changed so as to return an error when deleting a non-existent ACL token. This means that if Nomad successfully deletes the token but fails to persist that fact, it will get stuck trying to delete a non-existent token forever. Update the token deletion function to ignore "not found" errors and treat them as successful deletions. Fixes: #17833
Nomad version
v1.5.6
Issue
We were doing DR testing in one of our testing environments and did a snapshot restore to a 1 hour old consul snapshot. Since then, nomad has been trying to revoke SI tokens from Consul that don't exist:
Consul is generating lots of
Cannot find token to delete
errors as well.These are the same tokens that are failing to delete (total attempts for each URL over the last 3 days):
Questions:
The text was updated successfully, but these errors were encountered: