Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nomad-Consul sync for background SI token reconciliation #6719

Open
shoenig opened this issue Nov 18, 2019 · 1 comment
Open

Nomad-Consul sync for background SI token reconciliation #6719

shoenig opened this issue Nov 18, 2019 · 1 comment
Labels

Comments

@shoenig
Copy link
Member

shoenig commented Nov 18, 2019

As part of #6701 Nomad Servers will need to be able to perform background Service Identity token reconciliation.

There is an inherent race condition where a Nomad Server successfully requests a new SI token from Consul, but then crashes / loses leadership before it's able to persist the details of that token into the raft log. This isn't particularly bad for Nomad, which can simply retry the request later on with a new leader, but it could cause orphaned tokens to accumulate in Consul. Since SI tokens are not periodic and will have no TTL, such tokens would linger forever. To avoid that, Nomad Servers can periodically request from Consul a list of every token it knows about, and compare that with every SI token a Nomad cluster knows about. If there are any SI tokens in Consul generated for the Nomad cluster (compared with metadata stored in the Description field), that Nomad is not aware of, request Consul to revoke them.

@shoenig shoenig added type/enhancement theme/consul/connect Consul Connect integration labels Nov 18, 2019
@shoenig shoenig added this to the 0.10.3 milestone Nov 18, 2019
@shoenig shoenig self-assigned this Nov 18, 2019
@shoenig shoenig modified the milestones: 0.10.4, 0.11.0, unscheduled Feb 13, 2020
@shoenig shoenig removed their assignment Jul 10, 2020
@tgross tgross added this to Needs Roadmapping in Nomad - Community Issues Triage Feb 12, 2021
@tgross tgross removed this from the unscheduled milestone Feb 12, 2021
@tgross tgross removed this from Needs Roadmapping in Nomad - Community Issues Triage Mar 4, 2021
@tgross
Copy link
Member

tgross commented Nov 1, 2023

Starting in Nomad 1.7.0-beta.1 we've deprecated the workflow where the servers mint SI tokens and need highly-privileged Consul tokens. That workflow will be removed in Nomad 1.9.

Ideally by that point we'll have some Consul-side sync (as Consul has for k8s). I'm going to keep this issue open in the meantime but rename it slightly.

@tgross tgross changed the title Enable Nomad Servers to perform background SI token reconciliation Nomad-Consul sync for background SI token reconciliation Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants