-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"Nvidia GPU Device Plugin" not working #14888
Comments
I see the same behavior on 1.4.1. You can also quickly test it with the following job (which will not be planned if the GPU is not detected properly): job "gpu-test" {
datacenters = ["dc1"]
type = "batch"
group "smi" {
task "smi" {
driver = "docker"
config {
image = "nvidia/cuda:11.0.3-base-ubuntu20.04"
command = "nvidia-smi"
}
resources {
device "nvidia/gpu" {
count = 1
}
}
}
}
} |
|
It seems I smuggled an additional n into the image name (will fix it now), but on 1.3.2 the job is starting, on 1.4.1 it is not placed, thus the same behavior you observed. edit: I just realized, maybe we had a misunderstanding. My idea was to add a minimal working example for the maintainers to reproduce the problem. |
I have exact issue with 1.4.1, but 1.3.1 works fine |
have same problem with Nomad 1.4.1 , did you managed to run GPU job some how? |
No, we downgraded the GPU clients back to 1.3.5 and only run the servers on
1.4.1.
…On Wed, Oct 26, 2022, 10:06 Fr0stoff ***@***.***> wrote:
It seems I smuggled an additional n into the image name (will fix it now),
but on 1.3.2 the job is starting, on 1.4.1 it is not placed, thus the same
behavior you observed.
edit: I just realized, maybe we had a misunderstanding. My idea was to add
a minimal working example for the maintainers to reproduce the problem.
have same problem with Nomad 1.4.1 , did you managed to run GPU job some
how?
Thanks.
—
Reply to this email directly, view it on GitHub
<#14888 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOAOD6PDJCT7MESSUYAJIDWFDRATANCNFSM6AAAAAAREFYDV4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Struggling with the same issue on 1.4.2 fwiw. |
Seeing this on 1.4.2 as well. |
I have the same problem when upgrading from 1.3.5 to 1.4.1. |
Hi folks, we've seen this issue and it's on our pile to triage. If you use the reaction 👍 on the top-level post that's more helpful, unless you're seeing the problem on a different driver version than the original post. Does anyone have debug-level logs from a client during startup for this issue? It'd help kick off our investigation to see if the problem is in fingerprinting the device or whether the problem is in communicating with the driver. |
I can generate debug level logs tomorrow.
…On Wed, Nov 2, 2022, 14:00 Tim Gross ***@***.***> wrote:
Hi folks, we've seen this issue and it's on our pile to triage. If you use
the reaction 👍 on the top-level post that's more helpful, unless you're
seeing the problem on a different driver version than the original post.
Does anyone have debug-level logs from a client during startup for this
issue? It'd help kick off our investigation to see if the problem is in
fingerprinting the device or whether the problem is in communicating with
the driver.
—
Reply to this email directly, view it on GitHub
<#14888 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOAODYWWPUHDKHTDDSKMRDWGJQVHANCNFSM6AAAAAAREFYDV4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I've compiled a debug log during startup, let me know if this helps: https://gist.github.com/heipei/6d71b12fa086486b907729763981f27c |
Thanks @heipei. I've extracted the relevant log lines here: device plugin logs
It looks like the plugin is having trouble fingerprinting during the initial startup, but it's succeeding later (enough for the scheduler to detect that the client has done so, at least). I know it's been a minute since we did a Nvidia driver release, so I took a look at the repo and was reminded of hashicorp/nomad-device-nvidia#6. There hasn't been a release of the changes made there. @shoenig you noted in the PR that we had some fixes to do with the implementation -- do you recall what the symptoms were there? (If not, I can try to stand up a box on AWS with an Nvidia card and dig in further.) |
IIRC what #6 uncovered was that the external plugin still imported the nvidia stuff from |
Hi @tgross , I encountered the same problem today, by researching the code, I found that the Devices was not successfully updated in batchFirstFingerprints, maybe I can help fix it, so I made a PR, can you review it when you have time? |
Thanks for investigating and the PR @vuuihc! Fix should go out in the next releases of 1.4.x, 1.3.x, and 1.2.x |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.3.6+
Operating system and Environment details
CentOS Stream 8
Plugin "nomad-device-nvidia" v 1.0.0 (https://releases.hashicorp.com/nomad-device-nvidia/1.0.0/nomad-device-nvidia_1.0.0_linux_amd64.zip)
NVIDIA-SMI 515.57
Driver Version: 515.57
CUDA Version: 11.7
Issue
"Nvidia GPU Device Plugin" not working on Nomad v1.3.6+
Reproduction steps
Install plugin nomad-device-nvidia on nomad v1.3.6+
Config file:
Expected Result
There is a GPU in the output of the "nomad node status " command
![135](https://user-images.githubusercontent.com/52203136/195584474-344bfdc6-a322-49da-aace-3335832f2f3b.png)
1.3.5:
Log 1.3.5:
nomad1.3.5.log
Actual Result
There is no GPU in output of "nomad node status " command
![136](https://user-images.githubusercontent.com/52203136/195584676-179e0de3-7984-42fb-8a59-5c96c96c66b3.png)
Log 1.3.6:
nomad1.3.6.log
Thank you!
The text was updated successfully, but these errors were encountered: