Allow overriding evaluation priority for job commands and APIs

[schmichael]

Proposal

Jobs are dequeued by scheduler workers in priority order. Priority also affects preemption and may affect other Nomad behaviors in the future (it is intentionally documented broadly so that it can be used whereever job priorities need to be weighed ... for better and worse).

As of Nomad 1.1.6 Job Registration evaluations are given the priority of the Job, but Deregistrations (nomad job stop) uses the default priority of 50.

#11426 changes the latter to also use the priority of the job for deregistrations to match user expectations.

As Luiz proposed in a comment ideally an operator would be able to override the evaluation's priority when submitting or stopping a job.

This issue proposes allowing setting the Evalutation.Priority field directly from the CLI and API:

nomad job run  -eval-priority=40 abc.nomad
nomad job stop -eval-priority=95 abc

Use-cases

Nomad can have a large backlog of evaluations in busy clusters or when incidents occur. Operators have near-0 control over how that backlog is processed which can be tremendously frustrating when a high priority job seems "stuck" behind lower priority work because an operator attempted to stop it.

Since Nomad uses the Job.Priority field for preemption, it's also currently impossible to change scheduling priority without affecting preemption. Especially during incidents this is undesirable: an operator may have no idea how setting Job.Priority = 100 affects preemption and just wants to get the job evaluated as quickly as possible.

Attempted Solutions

There are no workarounds for controlling evaluation priority or much at all about evaluations.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.