Error on example model

[denix56]

🐛 Bug

I receive on every GAN model that I try to execute.

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
making sure all `forward` function outputs participate in calculating loss. 
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

I tried compete reinstall of pytorch in different conda environment, tried both 1.8.0 and 1.9.0.dev
I get error on both GAN example and my custom model.
My custom model has only 1 parameter (each of the networks), that I multiply with input, but it gives error too.

To Reproduce

GAN model from examples

Expected behavior

Environment

PyTorch 1.8.0 (or latest dev)
PyTorch-Lightning 1.2.3

Additional context

I run on 2 Nvidia V100 gpus in ddp mode

[awaelchli]

Fixed by #6460?

[awaelchli]

python pl_examples/domain_templates/generative_adversarial_net.py --num_processes 2 --accelerator ddp_cpu

fails on 1.2.3
works on master

fix will be released as part of the bug release this week

workaround for you until then:

Trainer(..., plugins=[DDPPlugin(find_unused_parameters=True)])

[denix56]

@awaelchli
I have lightning 1.2.5 and the bug still exists

[MarsSu0618]

Hi, @awaelchli
I encounter the same problem.

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable). (prepare_for_backward at /opt/conda/conda-bld/pytorch_1579022060824/work/torch/csrc/distributed/c10d/reducer.cpp:514)

My environment:

pytorch: 1.4
pytorch-lightning: 1.1.1

So can i upgrade version to solve it?

[tchaton]

Dear @denix56, @MarsSu0618,

Would you mind sharing a reproducible example for us to debug this behaviour ?

And please, update to PyTorch 1.6 / master on Lightning.

Best,
T.C

[Borda]

Hi @denix56, @MarsSu0618 could you check with the latest 1.2.6?
Also, I would not recommend relying on PT 1.9dev as it is not a stable release

[awaelchli]

Would it make sense to catch this PyTorch warning and replace it with a Lightning-friendly warning?
For PyTorch users it is clear that one has to set the flag in DDP but for Lightning users this is hidden away, and they can only know by going to our documentation.

We can catch the warning in forward in the wrapper, then replace it with a simple instruction to set the plugin parameter.
What do you think @SeanNaren @ananthsub @tchaton (tagging some random people who may know what I'm talking about?)

[SeanNaren]

Considering find_unused_parameters is now True by default, this should solve the issue here right @awaelchli?

There is a warning printed now since find_unused_parameters is set to True by default, which suggests to people to turn it off. If we could catch this and tell people to use accelerator=ddp_find_unused_parameters_false instead that would be great (once #7224 is merged) :)

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!