-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluation broker control API #11638
Comments
Some eval control would be nice. I have an eval scheduled for the future:
I've resolved the issue that caused this allocation to fail and would like a way to unblock re-deploying the job. Currently, there's no way to purge this pending eval, re-submitting the job is a no-op, setting |
Closing this issue as eval broker pause/un-pause as well as eval delete will ship in the next release. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
The Nomad engineering team would love to get some community feedback on a proposed set of features!
During incident response for Nomad or for Nomad workloads, operators may find that the scheduler can compound the ongoing incident by pushing forward with evaluations as best it can. We've recently been working on a bunch of improvements for incident response like bypassing
shutdown_delay
(#11448), makingnum_schedulers
tunable via API (#11449), overriding evaluation priority on job registration (#11434), and an API to halt job registration (#11450).Operators may want to have some sort of top-level "maintenance mode" where evaluation processing is paused or otherwise controlled while a problem is being debugged.
Eval Broker Pause/Resume
Evaluations are the unit of work for the Nomad scheduler. They are created...
The eval broker is the component on the leader that takes evaluations that have been written to raft and queues them for scheduling on one of the scheduler workers. Pausing the eval broker would prevent the scheduler queues from receiving new work and let the workers catch up so they can reduce CPU, memory, and disk resources. But note that this also includes any scheduling work necessary to reschedule failed allocations and periodic tasks! This isn't something to be done lightly, but as an emergency measure by operators.
For implementing Pause/Resume, there's an existing
enabled
flag which gets toggled whenever a server steps up/down from leadership (the eval broker only runs on the leader). We check this flag in a few places:enabled = true
(reffsm.go#L746
)enabled
flag (refeval_broker.go#L374-L377
)leader.go#L491-L517
)leader.go#L781
)So if we write a new Scheduler Configuration entry to raft, then whenever we get that RPC or whenever a server assumes leadership, it can check that value and call
SetEnabled
only if the eval broker should be enabled.Evaluation Purge/Delete
Another idea that's been considered is a
nomad eval purge
command, but this has a few sharp edges. If we flush the eval broker's queue, it'll immediately get filled back up again by the eval restorer on the leader. But if we delete all the pending/blocked evals from the state store, what happens to those that are in flight? So I think to do this safely we'd need to lock the eval broker (pausing it), find all the evals on the queue, delete them from the state store, and then flush the queue before unlocking it (unpausing it). That'll leave whatever evals are still in-flight in the scheduler, and any new evals/reblocks that come from the scheduler will block onEnqueue
because we're holding the lock.Another option (or perhaps in addition?) would be to only allow deleting a single eval with a command like
nomad eval delete :eval_id
. If we ensure that nil evaluations are handled safely, we could delete the eval from the state store and the eval broker's queue, and it would be a no-op for the scheduler. This would be useful in the case where a particular job's evals are a "poison pill" that generate high scheduler workloads (ex. a job with a very large jobspec or dispatch payload).Evaluation Force/Priority
A third idea we've considered is the option to force an evaluation to be bumped in the priority queue so that it's being evaluated ahead of other evaluations. This could be a
nomad eval priority
command that updates thePriority
field and raft indexes and forces it to be re-enqueued. A hypotheticalnomad eval force
command would be the same thing except less fine grained; it would simply update to the highest priority.The text was updated successfully, but these errors were encountered: