-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true #11052
Comments
Thank you for the report @madsholden. Just to highlight one point that you mentioned:
This seems like an important clue and would explain whey you see this intermittently. |
I just want to say that this is happening at my company too. I ran into this issue where I have a job scheduled to run every half hour with This seems like a high priority flaw. |
For reference, this was the schedule I was using: Also, madsholden said he sheduled their job to run every half hour, but based on the cron schedule they shared in their post, it looks like they scheduled it to run once per hour, on minute x:30. |
Same symptom here (Nomad 1.0.4) -- a duplicate instance of a non-overlapping periodic job got started during a leadership election. Not on the cron schedule, either.
|
I'm seeing the same problem on Nomad v1.1.0 - duplicate periodic process launched during leader election |
I've heard of a similar issue being resolved for some users by removing If anybody tries this, please let me know if it alleviates the issue. This doesn't fix the core issue of course, but might be a serviceable workaround until we are able to fix the root cause. |
This bug is still causing us problems from time to time. Is anyone actually working on trying to fix it? |
Hey @cread, sorry about this one dragging out. This is on our radar, but unfortunately not something we can prioritize right now. Between 1.4 commitments and some other pressing bugs, we're more or less at capacity for at least several weeks. I'll let you know if we pick it up. Or if anybody wants to take a crack at a fix, let me know and I can provide some rough guidance. |
FWIW - we're also seeing this issue with |
How is this not a higher priority issue? It seems like one of the main features of Nomad is to orchestrate jobs for you, yet it's not reliable and you have to orchestrate it yourself anyway. At the very least, you need to update your documentation so that people know that prohibit_overlap doesn't actually work. You're still advertising the feature in your documentation here: |
Referencing #14505 as a possible duplicate issue. |
Hi everyone, I am currently looking into this and as a first step have worked on a reliable local reproduction. I am using the Vagrantfile in the repository root for my isolation and am building Nomad using the current main source at Start a Nomad cluster using the cluster.sh script ensuring all processes are running as root via Register the jobspec below via job "test" {
type = "batch"
periodic {
cron = "* * * * * *"
prohibit_overlap = true
}
group "test" {
task "test" {
driver = "raw_exec"
config {
command = "sleep"
args = ["30000s"]
}
}
}
} Discover which Nomad server is the leader by using Upon transition, there will now be two instances of the periodic job running, despite the job including the
I will be working through this scenario to try and identify what behaviour is causing this, and will post any relevant findings in a followup comment. Thanks for everyones patience. |
…hibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous childre.
…hibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous children.
…hibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous childre.
…hibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous children.
…hibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous childre.
…hibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous children.
…hibit_overlap is true (#16583) * Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous childre. * Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous children. * style: refactor force run function * fix: remove defer and inline unlock for speed optimization * Update nomad/leader.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * style: refactor tests to use must * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * fix: move back from defer to calling unlock before returning. createEval cant be called with the lock on * style: refactor test to use must * added new entry to changelog and update comments --------- Co-authored-by: James Rasell <jrasell@hashicorp.com> Co-authored-by: James Rasell <jrasell@users.noreply.github.com>
…hibit_overlap is true (#16583) * Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous childre. * Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous children. * style: refactor force run function * fix: remove defer and inline unlock for speed optimization * Update nomad/leader.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * style: refactor tests to use must * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * fix: move back from defer to calling unlock before returning. createEval cant be called with the lock on * style: refactor test to use must * added new entry to changelog and update comments --------- Co-authored-by: James Rasell <jrasell@hashicorp.com> Co-authored-by: James Rasell <jrasell@users.noreply.github.com>
…hibit_overlap is true (#16583) * Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous childre. * Multiple instances of a periodic job are run simultaneously, when prohibit_overlap is true Fixes #11052 When restoring periodic dispatcher, all periodic jobs are forced without checking for previous children. * style: refactor force run function * fix: remove defer and inline unlock for speed optimization * Update nomad/leader.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * style: refactor tests to use must * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * Update nomad/leader_test.go Co-authored-by: James Rasell <jrasell@users.noreply.github.com> * fix: move back from defer to calling unlock before returning. createEval cant be called with the lock on * style: refactor test to use must * added new entry to changelog and update comments --------- Co-authored-by: James Rasell <jrasell@hashicorp.com> Co-authored-by: James Rasell <jrasell@users.noreply.github.com>
Nomad version
Nomad v1.1.0 (2678c36)
Operating system and Environment details
Ubuntu 20.04.2 on EC2
Issue
I have a periodic job set to run every half hour, with prohibit_overlap set to true. It is usually finished in much shorter time, but now it's running for longer, which is expected. However, Nomad has started three instances of this job. When checking the Nomad server logs I noticed both times when it started a new job, the servers were in the middle of a server election.
Reproduction steps
I can't reproduce it, it usually doesn't start a new instance when one is running.
Expected Result
Nomad should never run two instances of the same job.
Actual Result
Three instances of the job is running.
Job file (if appropriate)
Only relevant parts included. Please let me know if you want more of the job file.
Nomad Server logs (if appropriate)
From the Nomad UI:
Aug 03, '21 01:33:58 +0200 | Received | Task received by client
Aug 12, '21 03:11:02 +0200 | Received | Task received by client
Server logs around those times:
The text was updated successfully, but these errors were encountered: