-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot set GPUs using UUID as constraint #18112
Comments
Hi @ruspaul013! Constraints only inform the scheduler to place the allocations on a client that has those GPU IDs in its fingerprint. Once the allocation has been placed, it's up to the task driver / device driver to select specific GPU devices on the host. Can you confirm that the client your workloads are being placed on has the UUID you want, but that there just happen to be other GPUs on the same host and the selection is apparently random among them? |
Hello @tgross. Thank you for the reply! Yes, the placement of the jobs are made on the client I want. |
Ok, it looks like the Reserve API for the device drivers should be able to pass a device ID down to the device driver. But we're probably not doing that in the client's device hook. Let me have a chat with some folks internally about what the right design is there. |
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
Ok, I've been able to reproduce this fairly easily and discovered that the problem is actually in the scheduler and not in the client's device hook (which is good because that's much easier to fix!). The scheduler's To reproduce, I used the following plugin config for our plugin "example-fs-device" {
config {
dir = "/home/tim/fake-devices"
list_period = "1s"
unhealthy_perm = "-rw-rw-rw-"
}
} With 15 "device instances":
And using the following jobspec: jobspecjob "example" {
group "web" {
network {
mode = "bridge"
port "www" {
to = 8001
}
}
task "http" {
driver = "docker"
config {
image = "busybox:1"
command = "httpd"
args = ["-vv", "-f", "-p", "8001", "-h", "/local"]
ports = ["www"]
}
resources {
cpu = 128
memory = 100
device "nomad/file/mock" {
count = 1
constraint {
attribute = "${device.ids}"
operator = "set_contains"
value = "d50d2264-309e-11ee-b684-6b2b671ed775,daaab330-309e-11ee-99af-c7712ad76156"
}
}
}
template {
data = "<html>hello, world</html>"
destination = "local/index.html"
}
}
}
} This will fail to get the correct device IDs most of the time (depending on random selection of the 15 devices). With the patch in #18141 this now works reliably:
|
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
Thank you @tgross! I will try in the next days a local build of nomad with the changes you made. |
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
When the scheduler assigns a device instance, it iterates over the feasible devices and then picks the first instance with availability. If the jobspec uses a constraint on device ID, this can lead to buggy/surprising behavior where the node's device matches the constraint but then the individual device instance does not. Add a second filter based on the `${device.ids}` constraint after selecting a node's device to ensure the device instance ID falls within the constraint as well. Fixes: #18112
#18141 has been merged and will ship in the next regular release of Nomad 1.6.x, with backports to 1.5.x and 1.4.x |
Nomad version
nomad server:
Nomad v1.5.6
nomad client:
Nomad v1.5.6
Operating system and Environment details
Plugin "nomad-driver-podman" v0.5.0
Plugin "nomad-device-nvidia" v1.0.0
Issue
As it was mention in #15455, I could use UUIDs of GPU as attribute to set constraints. But when I want to use specific GPUs based on their UUID, I still get random GPUs.
Reproduction steps
Create a job file where you use multiple GPUs and set constraint based on their UUID.
Expected Result
running
nvidia-smi -L
inside container and see GPUs with the same UUID from job file.Actual Result
running
nvidia-smi -L
inside container and get random GPUs.Job file (if appropriate)
results:
The text was updated successfully, but these errors were encountered: