-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Not showing warning when using a taken GPU #18364
Comments
@ruspaul013 I'm surprised that job spec doesn't simply return a validation error, but the |
Hello @tgross, thank you for the suggestion. I tried it, but now I get an error even if I do not use
and I get this warning
I know that constraint block does not belong there, but the example from |
Yeah, that's something I often forget but you can set But setting the constraint only filters out nodes from scheduling. Taking a look at the code I think the problem is that Devices are not considered comparable resources, so they're not filtered here: Lines 172 to 178 in e3c8700
And this part of the code only looks for device oversubscription among the allocs being scheduled, so it ignores the ones already running in the node: Lines 201 to 207 in e3c8700
|
Nomad version
nomad server:
Nomad v1.6.1
+ patch with #18141nomad client:
Nomad v1.6.1
+ patch with #18141Operating system and Environment details
Plugin "nomad-driver-podman" v0.5.1
Plugin "nomad-device-nvidia" v1.0.0
Issue
I created a patch with the solution provides in #18141 to test on our cluster. While testing I discovered that if a GPU or multiple GPU are already used in jobs, nomad will not give a warning about this and will place the job without using those GPUs.
Reproduction steps
Expected Result
Throw an warning like
WARNING: Failed to place all allocations.
Actual Result
Place the job on the client.
Job file (if appropriate)
Job file 1:
Job file 2:
The text was updated successfully, but these errors were encountered: