core: reschedule evicted batch job when resources become available

[shoenig]

This PR fixes a bug where an evicted batch job would not be rescheduled
once resources become available.

Intuition: previously the scheduler would filter an alloc that was evicted but its tasked completed successfully. The problem is that that tasks were stopped because of the eviction stop signal, not because they had run to completion. Such an alloc needs to be queued to run again.

Closes #9890

[shoenig]

using the same test setup in #9890

➜ nomad job status low
ID            = low
Name          = low
Submit Date   = 2022-06-02T13:32:47-05:00
Type          = batch
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost  Unknown
group       0       0         3        0       1         0     0

Allocations
ID        Node ID   Task Group  Version  Desired  Status    Created  Modified
af37bf7d  81e51b03  group       0        run      running   11s ago  8s ago
0f968375  81e51b03  group       0        evict    complete  59s ago  34s ago
3cc484a6  81e51b03  group       0        run      running   59s ago  55s ago
af8f1ff6  81e51b03  group       0        run      running   59s ago  55s ago

[tgross]

LGTM! Nice debugging work!

[Fuco1]

Thanks team! This is really useful for some of our workloads. Great change!

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.