Pending batch jobs not run when resources become available #17278
Comments
mikenomitch
added
type/bug
theme/batch
|
I was not able to reproduce this with the following script and jobs, against a single agent in both server/client mode: repro.sh#!/bin/bash
set -x
# check nomad version
nomad version
curl -sS -H "X-Nomad-Token: $NOMAD_TOKEN" "$NOMAD_ADDR/v1/agent/self" | jq .config.Version
# issue menions namespace, so set one up
nomad namespace apply 'my-ns'
export NOMAD_NAMESPACE='my-ns'
# take up all the cpu
nomad run cpu-eater.nomad.hcl
# run and dispatch parameterized job,
# will sit pending due to cpu-eater
nomad run batch.nomad.hcl
nomad job dispatch bat
# i tried running a bunch, but got the same result.
#for _ in {1..20}; do
# nomad job dispatch -detach bat
#done
# free up cpu
nomad stop -detach cpu-eater
# wait for pending job to run
for _ in {1..3}; do
nomad status bat | grep pending || break
sleep 10
done
# confirm 'hello' from batch job output
nomad logs -job $(nomad status bat | awk '/dead/ {print$1}' | head -n1)
# cleanup
nomad stop bat
nomad system gccpu-eater.nomad.hcljob "cpu-eater" {
datacenters = ["dc1"]
namespace = "my-ns"
group "g" {
task "t" {
resources {
cpu = 92000 # my machine is pretty beefy.
}
driver = "docker"
config {
image = "busybox:1"
command = "sleep"
args = ["infinity"]
}
}
}
}batch.nomad.hcljob "bat" {
datacenters = ["dc1"]
namespace = "my-ns"
type = "batch"
parameterized {}
group "bat" {
task "bat" {
driver = "docker"
config {
image = "busybox:1"
command = "echo"
args = ["hello"]
}
}
}
}
resulting output shows that the pending batch job did get run: Perhaps I'm missing a component of the replication steps? |
|
Working with @louievandyke today, we narrowed down the extra piece required to reproduce this, all the way up to current latest Nomad Not only do we need a namespace, but that namespace needs a quota on it (an enterprise feature), specifically a quota where the resource (cpu, in this case) is set to zero Modifying the namespace piece of nomad quota apply quota.hcl
nomad namespace apply -quota 'my-quota' 'my-ns'where name = "my-quota"
limit {
region = "global"
region_limit {} # default zero everything
}results in the dispatched job getting stuck pending: $ nomad status
ID Namespace Type Priority Status Submit Date
bat my-ns batch/parameterized 50 running 2023-10-18T16:35:00-05:00
bat/dispatch-1697664900-14f2784a my-ns batch 50 pending 2023-10-18T16:35:00-05:00
cpu-eater my-ns service 50 dead (stopped) 2023-10-18T16:34:48-05:00Specifically, this is due to the eval getting There will be a server log entry like
The $ nomad eval list | grep blocked
ec85a56a 50 queued-allocs bat/dispatch-1697664900-14f2784a my-ns <none> blocked N/A - In Progress
These metrics are pretty neat too, and show that the reason in this case is CPU (this can also happen with memory) $ nomad operator metrics -format=prometheus | grep ^nomad_nomad_blocked_evals
nomad_nomad_blocked_evals_cpu{datacenter="dc1",host="cool-host",node_class=""} 100
nomad_nomad_blocked_evals_job_cpu{host="cool-host",job="bat/dispatch-1697664900-14f2784a",namespace="my-ns"} 100
nomad_nomad_blocked_evals_job_memory{host="cool-host",job="bat/dispatch-1697664900-14f2784a",namespace="my-ns"} 0
nomad_nomad_blocked_evals_memory{datacenter="dc1",host="cool-host",node_class=""} 0
nomad_nomad_blocked_evals_total_blocked{host="cool-host"} 1
nomad_nomad_blocked_evals_total_escaped{host="cool-host"} 0
nomad_nomad_blocked_evals_total_quota_limit{host="cool-host"} 0I haven't quite tracked down yet why quotas with zero values produces this result, but I have confirmed that setting CPU in the quota results in jobs getting serialized appropriately based on the quota with a (temporary) message like I'll pick this back up tomorrow and try to narrow down where we might handle the zero value differently to avoid this. |
|
I believe I've found the cause. An evaluation getting blocked is totally normal when resources are occupied, but such evals should be _un_blocked automatically when the resources are freed up. So something is preventing them from being unblocked when the previous alloc(s) complete. Seems this here is the culprit: func (b *BlockedEvals) unblock(computedClass, quota string, index uint64) {
// .....
if quota != "" && wrapped.eval.QuotaLimitReached != quota {
// We are unblocking based on quota and this eval doesn't match
continue
// .....The if "my-quota" != "" && "" != "my-quota" { // don't unblock }
// ^ quota on the ns ^ quota that caused the block (none!)This can also happen when the quota is very large, like I have a tentative fix for it, will validate some more and write tests for it next week! |
Opening bug on behalf of a customer:
Description
We have observed Nomad 1.1.2+ent (with namepsaces) will put jobs into Pending state when not enough resources are available to execute them, however when resources become available, it will not automatically run those pending jobs. Kicking the scheduler via either marking any node as ineligible/eligible, or running
nomad job eval -force-reschedulewill kick it to life, and another single set of tasks will run. On completion, pending tasks will remain stuck.Example
The text was updated successfully, but these errors were encountered: