-
-
Notifications
You must be signed in to change notification settings - Fork 609
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workaround for torch.linalg lazy init for multi-GPU setup? #3004
Comments
@Nicholas-Autio-Mitchell seems like the original repository is using single gpu training with a single gpu (if they do not encounter any similar issues). In your case, most probably you have multiple gpus and you can launch the same training by exposing only a single cuda device, for example: CUDA_VISIBLE_DEVICES=0 python train.py -cn exp_kitti_360 If you would like to train on multiple gpus using DDP, you can try it using (I checked quickly the code and it looks like it should be able to parallelize) + add "backend: 'nccl'" into the config yaml file: torchrun --nproc_per_node=N train.py -cn exp_kitti_360 Hope this helps |
Thanks for your answer @vfdev-5. Using only a single GPU does work; I had already been training with a single GPU using I had not tried using
I have managed to find the unused model parameters and remove them (final layers of pre-trained backbone). Is there any way to simply allow unused parameters? |
@Nicholas-Autio-Mitchell I think you can pass |
Yes, that did work fine - as mentioned, I found and removed them so distributed training is working. I was wondering if it is possible to simply allow unused parameters? The setup is quite complex and I don't want to find and remove parameters for all training configurations and dataset variants. |
❓ Questions/Help/Support
I am new to the ignite library, and am trying to run multi-GPU (single node) training that is configured in this repo, and here is the specific entrypoint I am running where
ignite.distributed.Parallel
is called.The error I am facing seems quite generic and that suggests a setup issue:
I found this error and explanation on torch repo: pytorch/pytorch#90613. It suggests newer CUDA versions can handle it better and provides a hack: calling
torch.inverse(...)
at the entrpoint to force the lazy initialisation oftorch.linalg
to happen. This doesn't fix the problem for :(I don't want to get into the details of the other repo with this question, just first wanted to ask if there is a known workaround when using the
ignite
library (apart from upgrading CUDA)?Thanks in advance for any help!
Version info
The full environment given by the above repo: https://github.com/Brummi/BehindTheScenes/blob/main/environment.yml
Ubuntu 18.04.5 LTS
Driver Version: 470.182.03
The text was updated successfully, but these errors were encountered: