-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NOMAD_ALLOC_INDEX is not always unique within a single service job version #10727
Comments
Hi @dpn! The relevant text here from #6830 (emphasis added):
It's not clear to me from what you've provided what the state history is for the allocations where you're seeing the reused indexes. Especially given you rolled the clients, it's entirely possible you have allocations that got marked lost and were rescheduled, deployments kicked off, etc. Can you provide the jobspec (especially the
I'm glad you figured out a hackaround, but you really shouldn't be relying on the uniqueness of the allocation index. The allocation ID is a UUID and is the canonical way to refer to an allocation. |
Doing some issue cleanup and it looks like we never heard back on this one. Going to close it out. |
Sorry @tgross for some of these "fire and forget" reports. We're a little understaffed and I just haven't had the bandwidth to follow up on these :( |
Welp, our monitoring team escalated another reproduction of this issue to me so looks like this issue is back on the menu!
Yep, totally understood and this is what I've been telling our customers. Unfortunately it's a heavy lift for them to reconfigure every job which is relying the documented behavior1 so they're looking for an upstream fix. To kick things off, we're on a newer build than was originally reported, although the rest of the conditions still hold:
Status for the suspect job:
So, we have 3 allocations, no deployments in progress, the previous deployment went out successfully, no failed tasks for this or the previous job version in the history and the canaries are not enabled. However, when we inspect those 3 running allocations we see an allocation index of 0 is being reused which seems to conflict with the docs:
One other interesting tidbit we've discovered is that the allocations from the previous version of the job are displaying the same behavior:
The rest of the relevant information can be found in the following files: alloc-0b1a9d76.txt Hopefully this makes sense- please let me know if there's anything else I can gather. Thanks! |
Ok, reopening this as that's definitely not the correct behavior assuming no failures or canaries. Note that you are using a fairly old version of Nomad, one that's won't be getting backported bug and security fixes after 1.3.0 goes out in a couple months, so even if we figure out what's wrong here you may end up needing to upgrade to a newer version to get the fix (or backport the patch yourself).
Interesting. The output of |
Perfect, thank you so much! Yep we're aware of the old version- we're a few versions behind but we are upgrading at a quarterly cadence-- we have no expectations on you guys to provide a backport if a fix should becomes available. That deployment has been GC'd at this point, but I've had the team kick another and attached dumps from this one: alloc-5ea975aa.txt |
Judging by the version numbers on that |
Yep, nailed it. On the few jobs we've seen this on it seems that once the job is in this state it does continue for some time- I'm not sure if they've ever become "fixed", but I can dig a bit and maybe shed some light on that.
Seems plausible from here. We generate our specs via some internal tooling- hopefully the json rendering works for ya: Also the job inspect is attached to my earlier post if that helps fill in any gaps from this spec. |
Interestingly enough, the job from the original report is still in this state- so it's managed to survive a version upgrade:
This is a completely different job from the one we've been looking at over the past few days. It's running on a different Nomad cluster, although both of these clusters in question are federated together. Would you be interested in the same dumps for this one? Edit: And thanks for all of your help on this-- really appreciate your time. |
So interestingly @DerekStrickland and I were chatting about this bit of code for a project he's working on, and he noticed that the code that stops allocations prefers to pick ones with the highest "name index" but specifically has code to deal with the notion that we could end up with duplicate names here
One hypothesis I have is that if the task group count drops below the name index, it'll correct itself. But obviously that's not a good workaround in general and it's totally useless in this case because the problematic name index is
I think we're good so far with what you've provided, thanks! |
Ahh wonderful news! Thanks again for digging in we really appreciate it 🙏 |
I can confirm that this works. But it's very inconvenient workaround. |
I wanted to follow up on this. I think trying to whack-a-mole the problem isn't going to get us the results we want when the design itself isn't really equipped to solve this problem definitively. The only way to strictly enforce this would be to reject plans that have overlapping alloc index at the plan apply step. This would increase the rate of plan rejections, but could also make a few other features like per-alloc volumes for CSI operate more nicely. I'm going to mark this for further discussion and roadmapping. |
Hello everyone, I just wanted to provide an update to this issue as I have been spending a good amount of time on it recently and have made progress. Our first approach to reject job plans within the plan applier (linked PR) seemed fine from a code standpoint, but when we discussed it further internally we realised we could create un-schedulable jobs. This is an approach we could therefore not take, and this extends further to producing errors during the reconciliation or scheduling on duplicate allocation indexes. It is therefore required we solve this bug, rather than reject any occurrences we find. I have been working on understanding the code which has allowed me to create a reproduction of this error. I can now see how we get into this error state and hope to be able to locate the source of this bug soon. |
Nice to hear. After this fix #16401 I see this issue much rarer but still see in some cases. |
Nomad version
Operating system and Environment details
CentOS Linux 7 (Core)
3.10.0-1160.24.1.el7.x86_64
Issue
According to #6830,
NOMAD_ALLOC_INDEX
is supposed to be unique across job versions. However, we have discovered a case in one of our clusters where a job where allocs appear to be on the same job version and have duplicateNOMAD_ALLOC_INDEX
s:This was discovered by our Prometheus metrics exporter complaining about attempting to ship duplicate metrics. Of interest, this job has a task count of 50, and only 4 out of those 50 are duped:
Interestingly enough, we had the job owner redeploy in attempt to fix this situation and we actually see the exact same
NOMAD_ALLOC_INDEX
s duplicated:Reproduction steps
Man, I wish I could tell you. We just upgraded the cluster from 0.11.4 to 0.12.10 earlier today so it's probably related to that... We upgraded the server cluster, then rolling upgraded the clients.
My guess is that the deployment prior to the cluster upgrade failed, and that the rolling client restart caused alloc migrations which started up the replacement allocs on the "new" job version, butso far I haven't had the bandwidth to attempt a reproduction.Edit: Scratch that. Deployment history seems fine:
So we put a hack into our exporter to stop the bleeding for the night. But this also means I can grab logs or anything interesting that will help on your end! Let me know what you need and I'll try to get it!
Expected Result
NOMAD_ALLOC_INDEX
should be unique within a job versionActual Result
NOMAD_ALLOC_INDEX
is not unique within a job versionJob file (if appropriate)
Nomad Server logs (if appropriate)
Nothing appears relevant here
Nomad Client logs (if appropriate)
I didn't see anything relevant in here, but let me know if you'd like me to collect them.
The text was updated successfully, but these errors were encountered: