-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GPU resources #406
Comments
Hi Francis, I am interested to hear some use cases and applications that would run on Nomad and use GPUs. At the end of the day it's upto applications to exploit heterogenous processor architectures on a host. A cluster manager can help with applications which wants to use GPUs by placing them on a machine which has GPUs and accounting their usages and by being the resource allocator and schedule different jobs which wants GPU resources available in the cluster. We could potentially fingerprint a machine for GPU resources and expose them to users by the GoLang bindings of CUDA. But before we do anything with GPUs it would be nice to understand some use cases and applications which people are already running in production. — On Tue, Nov 10, 2015 at 2:39 PM, Francis Chuang notifications@github.com
|
Some possible use-cases:
|
@F21 I think we can support this via fingerprinting, and in AWS this can probably already be detected via instance type. I'm not sure what kernel support exists for this (I suspect none without custom drivers) but unless there is some way for Nomad to monitor how many GPU resources a process is using I don't think it makes much sense in Nomad core. Deeper integration, e.g. with the CUDA API or similar, will require a custom builds with a C toolchain. I think this is a good candidate for a feature that can be implemented as a plugin once we have a plugin architecture in place. In that case we'd want support for Nomad to schedule based on plugin-defined resources. |
@cbednarski I think once we have a plugin architecture in place, we should be able to have fingerprints outside our main binary to enable such use cases. My main question was around how do we expose the GPU capacity available on a node and if there are any ways to bound resources across multiple Tasks using the available GPU on a node. |
@diptanu Yeah that's the part where I think this requires integration with the CUDA API, or use of a customized kernel. There's a lot of research in this area but I'm not aware of any mainline features to support this. Assuming there is a way, though, I think it should be possible to support some kind of arbitrary resources in the scheduler. Likely we will want this to track other constrained or provisioned resources like block storage, ENI, SDN, etc. |
Yes, custom resources would be hugely useful |
There is no need to implement exposing of gpu. Let's just allow to define custom resources and allow custom drivers to handle them. It is connected to #1061 |
+1 for GPU resources |
As of Nomad 0.9 which is currently available as a beta, we have support for scheduling jobs that require access to arbitrary devices, and a default plugin which provides access to |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Mesos recently announced some work they have been doing with nvidia to treat GPU resources like CPU and memory resources: https://mesosphere.com/blog/2015/11/10/mesos-nvidia-gpus/
It would be interesting to see if this is something nomad can support too.
The text was updated successfully, but these errors were encountered: