Easy way to pick among multiple GPUs

[ViralBShah]

It would be nice to have an easy way to pick one among multiple GPUs.

[maleadt]

The CUDAnative part of this is implemented: device!(::CuDevice), with automatic initialization of the CUDA API for device 0 if the user doesn't pick anything. To make this really useful, CuArray would probably have to know about which device it is tied to, or alternatively use unified memory (this seems like a much easier route, and aligns with recent CUDA toolkit developments).
Demo: https://github.com/JuliaGPU/CUDAnative.jl/blob/master/examples/multigpu.jl

[CarloLucibello]

No news for this? demo link is broken

[maleadt]

https://juliagpu.gitlab.io/CUDA.jl/usage/multigpu/

[CarloLucibello]

Nice, didn't see that, thank you. So, on a gpu box I get this

julia> ngpus = length(CUDAdrv.devices())
3

julia> CUDAnative.device!(2)

julia> CUDAdrv.device()
CuDevice(2): GeForce RTX 2080 Ti

while on cpu-only one I get:

julia> CUDAdrv.devices() |> length
ERROR: could not load library "libcuda"
....

Is this than a reasonable way to write a Flux script?

gpu_id = 0  ## set < 0 for no cuda, >= 0 for using a specific device (if available)

if CUDAapi.has_cuda_gpu() && gpu_id >=0
    CUDAdrv.device!(gpu_id)
    CuArrays.allowscalar(false)
    device = Flux.gpu
    @info "Training on GPU-$(gpu_id)"
else
    device = Flux.cpu
    @info "Training on CPU"
end

model = model |> device
for x in data
    x = x |> device
    ....

If so, maybe importing 3 cuda packages is not super smooth for the user, we could wrap some of that functionality within Flux

[maleadt]

You can inspect CuArrays.functional(), but yeah, the CUDAapi functions are supposed to be the user friendly ones. Flux already has a gpu function that uploads to the GPU if one is available.
https://github.com/FluxML/Flux.jl/blob/069d22869313d2eb7dc04d64dc7c7a819643acf7/src/functor.jl#L108

[CarloLucibello]

We could add a Flux.gpu! setting the device and with an allowscalar optional argument, so that the script simplifies to

gpu_id = 0  ## set < 0 for no cuda, >= 0 for using a specific device (if available)

if CUDAapi.has_cuda_gpu() && gpu_id >=0
    device = Flux.gpu!(gpu_id, allowscalar=false)
    @info "Training on GPU-$(gpu_id)"
else
    device = Flux.cpu
    @info "Training on CPU"
end

[maleadt]

multi-gpu has greatly improved, so I think we can close this.