multi gpu slower than single setup #189
Comments
|
Hi, So the number of words, sentences/sec is per GPU. So if you run a 2 GPU jobs and each is printing 100 sent/s it means the actual training speed is 200 sent/s. In the ideal scenario, the increase in speed is linear in the number of GPUs, and you would see the same number of sent/s in the training log. But in your case it is slower in any case, so there must be something wrong. Does it generate 2 different folders? Or two different training log files? You should observe 2 training log files, but in the same folder, along with the Can you share the command you used to start your multi-GPU experiment? |
|
Thank you for explanation, as you said there was definitely a problem on my setup. Single GPU
And this is the logs which runs on single gpu
Multi GPU
It generates 2 different folders in dumped folder.
And here is the log outputs; nnk25v6kjt/train.log
n5t2wqqpfr/train.log-1
nvidia-smi shows full gpu util. on each cards and 2 different processes running on them: | 0 GeForce RTX 208... Off | 00000000:01:00.0 Off | N/A | | 0 3607 C .../dh/environments/pytorch-env/bin/python 5499MiB | |
|
Mmm this is weird, not sure I understand what is going on here. First thing to check would be to see whether the parameters are actually shared across GPUs. Could you try after each iteration step to do something like: |
|
@glample here it is dumped/exp06_2X_2080S/4t1fvd7v88/train.log
dumped/exp06_2X_2080S/rgy0ay2xj9/train.log-1
|
|
btw these cards are different branded. One of them is gigabyte and the other one is asus. But I don't think it will be caused by this |
|
This means that the parameters are properly synchronized between the two processes. It is usually a good thing, but here it means that everything is working as expected, so unfortunately I don't understand why the code is slow. I would imagine this is because the communication between your two GPUs is not fast enough. If the cards are different branded, this is maybe not surprising. What you can try is to use |
|
Hmm, |
|
oh crap! 1 x PCI Express x16 slot, running at x4 sorry for taking your time |
|
Hi, I have very similar problem with speed. CPUIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz MBSuperMicro X10DRG-Q 1x RTX 2080export CUDA_VISIBLE_DEVICES=0
export NGPU=1
python -W ignore::UserWarning -m torch.distributed.launch --nproc_per_node=$NGPU train.py \
--exp_name xlm_cs-en \
--dump_path ./dumped \
--data_path $OUTPATH \
--lgs 'cs-en' \
--clm_steps '' \
--mlm_steps 'cs,en' \
--emb_dim 1024 \
--n_layers 12 \
--n_heads 8 \
--dropout 0.1 \
--attention_dropout 0.1 \
--gelu_activation true \
--batch_size 4 \
--bptt 256 \
--optimizer adam,lr=0.0001 \
--epoch_size 300000 \
--max_epoch 100000 \
--validation_metrics _valid_mlm_ppl \
--stopping_criterion _valid_mlm_ppl,25 \
--max_vocab 150000 \
--amp 01 \
--fp16 true10x RTX 2080export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9
export NGPU=10
python -W ignore::UserWarning -m torch.distributed.launch --nproc_per_node=$NGPU train.py \
--exp_name xlm_cs-en \
--dump_path ./dumped \
--data_path $OUTPATH \
--lgs 'cs-en' \
--clm_steps '' \
--mlm_steps 'cs,en' \
--emb_dim 1024 \
--n_layers 12 \
--n_heads 8 \
--dropout 0.1 \
--attention_dropout 0.1 \
--gelu_activation true \
--batch_size 4 \
--bptt 256 \
--optimizer adam,lr=0.0001 \
--epoch_size 300000 \
--max_epoch 100000 \
--validation_metrics _valid_mlm_ppl \
--stopping_criterion _valid_mlm_ppl,25 \
--max_vocab 150000 \
--amp 01 \
--fp16 true \
--accumulate_gradients 4I tried printing this after every iteration and it returns same value for all gpus: logger.info(sum(p.sum().item() for p in model.parameters()))Also, I am getting this warning: Overall, it looks like training with 10x RTX 2080 is only a little bit faster than with 1x RTX 2080. I am getting very similar results for another server with 5x1080. I used Horovod for parallel training and it was always fine (not linear, but close enough). Is there something, that I can try? Thanks. |
Hi,
I am not sure that but it looks like slowed down when use 2 gpu cards.
When I setup single gpu, training goes with 220 sent./s
But after using multi gpu (2 cards) with same params, training slows down to 85 sent. /sec.
And it generates 2 different foldes under dumped folder.
Is it normal behaviour or do I miss something?
Thanks
The text was updated successfully, but these errors were encountered: