Fix ddp for test

[tullie]

What does this PR do?

DDP for Trainer.test is broken. The problem is that when each distributed process is spawn, at the start of the test function, a random port is set. When we get to the init_process_group part, because each process has a different port set, the program hangs.

These lines were introduced in https://github.com/PyTorchLightning/pytorch-lightning/pull/2512/files. I'm wondering why they were added in the first place?

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #2866 into master will decrease coverage by 2%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #2866    +/-   ##
=======================================
- Coverage      90%     88%    -2%     
=======================================
  Files          79      79            
  Lines        7472    7469     -3     
=======================================
- Hits         6756    6568   -188     
- Misses        716     901   +185     

[codecov]

Codecov Report

Merging #2866 into master will decrease coverage by 2%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #2866    +/-   ##
=======================================
- Coverage      90%     88%    -2%     
=======================================
  Files          79      79            
  Lines        7472    7469     -3     
=======================================
- Hits         6756    6568   -188     
- Misses        716     901   +185     

[Borda]

how does the fix work as previous tests were also passing, mind adding a test for the failure case?

[awaelchli]

@Borda This only affects ddp. Since we don't have ddp testing, things like this won't show up. I have a PR here that adds some basic ddp tests #2856

These lines were introduced in https://github.com/PyTorchLightning/pytorch-lightning/pull/2512/files. I'm wondering why they were added in the first place?

@tullie @williamFalcon explained to me that choosing random ports is necessary to avoid an error message saying "tried to init ddp connection multiple times" (paraphrasing) when you do forexample .fit and then .test, ultimately calls model.init_ddp_connection twice.
However, I think this is just a workaround and I tried to make it use the same connection without running init multiple times here in this PR #2790 .

[tullie]

Going to close this because seems like #2790 is the best wip diff to work on this. @awaelchli will try and look into more of these issues this weekend.

[awaelchli]

Thank you tullie. #2790 has lots of dirty debug code so apologies, I ran into some weird things that made me desperate :))