Docker driver mount device support / documentation

[KamilLelonek]

Nomad does not support (or does not document) how to mount devices on a Docker container.

A use case here being mounting the Nvidia GPUs to a docker container. The Docker syntax to do this is the --device switch. For example:

docker run --device=/dev/nvidia0:/dev/nvidia0 cuda

Can this support please be added, or, if already present, documented?

[dadgar]

Do you have use for this outside of GPUs? First class GPU support will come.

[KamilLelonek]

The only use is to access Nvidia GPU.

…

On Thu, 25 May 2017 at 17:50, Alex Dadgar ***@***.***> wrote: Do you have use for this outside of GPUs? First class GPU support will come. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2671 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADDRZZ85AW9dwTW9JyhMHxNn1GuYT96Sks5r9aNfgaJpZM4NmXpZ> .

[dadgar]

@KamilLelonek Okay I am going to close in favor of: #406.

This is being actively worked on!

[KamilLelonek]

What do you suggest for now? Should I use raw script with flags provided?

…

On Thu, 25 May 2017 at 17:53, Alex Dadgar ***@***.***> wrote: @KamilLelonek <https://github.com/kamillelonek> Okay I am going to close in favor of: #406 <#406>. This is being actively worked on! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2671 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADDRZYrzR_DsRFOkUDL8L1OONMIt4z9_ks5r9aQXgaJpZM4NmXpZ> .

[dadgar]

@KamilLelonek Yep that is the best route to take for the time being!

[schmichael]

Closing as a duplicate of #406

[KamilLelonek]

Do you think that for such case at the moment job.nomad should contain the following:

task "server" {
  driver = "exec"

  config {
    command = "docker run docker-registry/image-name:tag"
    args    = [
      "--device=/dev/nvidia0:/dev/nvidia0",
      "--device=/dev/nvidia1:/dev/nvidia1",
      "--device=/dev/nvidia2:/dev/nvidia2",
      "--device=/dev/nvidiactl:/dev/nvidiactl",
      "--device=/dev/nvidia-uvm:/dev/nvidia-uvm",
      "--device=/dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools"
    ]
  }
}

?

[dadgar]

I would do it with raw_exec but that should work. I would clean up the image afterwords!

[schmichael]

Just add `--rm` to cleanup the image on exit. And raw_exec will work much better. You'll probably also need to run the task as a user with access to docker. The easiest would be to just add the nobody user to the docker group, but that gives docker access to any number of services.

…

On Fri, May 26, 2017 at 1:33 PM, Alex Dadgar ***@***.***> wrote: I would do it with raw_exec but that should work. I would clean up the image afterwords! — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2671 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAG60hgoSI4Um79M4GGXHy3xGCbeAPLJks5r9zc2gaJpZM4NmXpZ> .

[KamilLelonek]

Thanks, I need to change it anyway because I must provide only executable and the rest of arguments as an array. On Fri, 26 May 2017 at 22:37, Michael Schurter <notifications@github.com> wrote:

…

Just add `--rm` to cleanup the image on exit. And raw_exec will work much better. You'll probably also need to run the task as a user with access to docker. The easiest would be to just add the nobody user to the docker group, but that gives docker access to any number of services. On Fri, May 26, 2017 at 1:33 PM, Alex Dadgar ***@***.***> wrote: > I would do it with raw_exec but that should work. I would clean up the > image afterwords! > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > <#2671 (comment)>, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AAG60hgoSI4Um79M4GGXHy3xGCbeAPLJks5r9zc2gaJpZM4NmXpZ > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2671 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADDRZVMSj7W1jgipR570RFB4fa4MyCOYks5r9zgLgaJpZM4NmXpZ> .

[schmichael]

Oops, I misspoke on --rm above. It should have been:

Just add --rm to cleanup the image container on exit. You'll need a script to also cleanup the image if desired.

[jaychris]

We need "--device" to mount usb devices and other serial control devices for hardware that container based applications are managing.

[pieterbreed]

I would like to have access to /dev/fuse inside my container that has an instance of riofs running. (An alternative to s3fs). To get this to work, I have to give both privileged and --device /dev/fuse on the docker command line. It would be nice if nomad supported devices in this way, over and above what #406 deals with.

[schmichael]

Allowing mounting arbitrary devices makes sense. I think the distinction between GPUs (#406) and other devices is we intend for the scheduler to have first class knowledge of GPU resources. Other devices will just supported in driver configs. Jay or Pieter: Would one of you mind opening a Github issue for your "--device" use cases? It shouldn't be blocked by #406.

…

On Mon, Jul 31, 2017 at 8:36 AM, Pieter Breed ***@***.***> wrote: I would like to have access to /dev/fuse inside my container that has an instance of riofs running. (An alternative to s3fs). To get this to work, I have to give both privileged and --device /dev/fuse on the docker command line. It would be nice if nomad supported devices in this way, over and above what #406 <#406> deals with. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2671 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAG60quoYRK_HU43-7g7RcEwhU7S5J4Dks5sTfRkgaJpZM4NmXpZ> .

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.