Memory used across all GPUs

[mys007]

The current implementation allocates hundreds of MB of GPU memory on each GPU present in the system (at least 102MB per device as reported by nvidia-smi), just upon simple require 'cutorch'. This doesn't change with subsequent calls like cutorch.setDevice(). Is there any technical reason for this behavior?

[soumith]

we only allocate a tiny amount of scratch space. All combined to be less than 2MB afaik.
NVIDIA's driver allocates a minimum amount of memory per process using the GPU, for p2p scratch buffers etc. (possibly).

[mys007]

Hmm, but the end effect is the same, there is 100MB GPU memory less for each process. It should be possible to start cutorch selectively just on a chosen subsets of GPUs.

[soumith]

CUDA_VISIBLE_DEVICES=0,2 th [yourscript.lua]

where you are telling it to use device 0 and device 2.
The devices are 0-indexed.

[mys007]

Great, thanks!

[eriche2016]

so if you use CUDA_VISIBLE_DEVICES=0 th [yourscript.lua], it means that you can only have 1 gpu to use and others are transparent to you, so there is no sense to use cutorch.setDevice(id) which can be used to switch the default gpu, right? if so, can you offer some guidlines about when to use CUDA_VISIBLE_DEVICES and when to use cutorch.setDevice(id)?

[fmassa]

@eriche2016 it makes sense to use it in the context of multi-GPUs. For example, with CUDA_VISIBLE_DEVICES=0,2 you select the GPUs you want to use (here 0 and 2), and with cutorch.setDevice you select between each visible device your tensor will be created. This is how it's done in DataParallelTable.
Now if you only want to use 1 GPU, then there are no benefits in using cutorch.setDevice instead of CUDA_VISIBLE_DEVICES.