[WIP] Fix Trainer.test in ddp before running Trainer.fit

[awaelchli]

What does this PR do?

Fixes #2683
Fixes #2765
Fixes #2807
Fixes #2537
(maybe also #2901)

trainer = Trainer(distributed_backend="ddp", gpus=2)
# no fit before
trainer.test(model)
# this hangs because master ports are different on rank 0 and rank 1

Core issues discovered here

Fixing this revealed a multitude of interconnected issues:

• Selecting a random port does not work for 2 main reasons. (a) determinism in subprocess not easy, (b) if you run fit many times, you eventually run out of ports
• In ddp, each subrocess excluding rank 0 has to init ddp with the same master port. When we run fit and then test, it has to happen twice, because between fit and test the subrocesses must be killed and restarted except rank 0. Unfortunately we cannot reuse the connection on rank 0, becaues the other new processes will see it as "port still in use", we have to create a new one on a new port.

Solution:

1. Find a free port (not random) on rank 0
2. broadcast the port number through torch.distributed and let subprocesses init their connection (and rank 0 as well)
3. Finally make sure we properly destroy the connections so that we never run out of ports

TODO:

• See what's going on in pytest / CI
• Ask someone nicely if they could try it on SLURM

[awaelchli]

@williamFalcon This fixes the reported issue, but I am afraid this causes other problems. In which case do we need to select the port randomly (by force)? And why is it only doing that in test and not fit?

[williamFalcon]

the port needs to be randomly chosen when running on a single node. But it should use the same port for test and fit

in the common case,
fit (sets port)
test (should not set port again)

in this case
no fit
test (port not set, then set it)

[awaelchli]

I just don't understand the force part. Different ranks will get different master port if we set force=True.

[awaelchli]

didn't find a way to write a test case, since this requires ddp for testing. any ideas?

[awaelchli]

@williamFalcon found this in the code

PID = os.getpid()
RNG1 = np.random.RandomState(PID)
RANDOM_PORTS = RNG1.randint(10000, 19999, 1000)

in DDP, pid will be different for each gpu, so it will yield different master ports when force=True. It looks like this is the root of the issue.

[pep8speaks]

Hello @awaelchli! Thanks for updating this PR.

• In the file pytorch_lightning/accelerators/ddp_backend.py:

Line 270:13: E265 block comment should start with '# '
Line 275:13: E265 block comment should start with '# '
Line 275:13: E303 too many blank lines (2)

• In the file pytorch_lightning/core/lightning.py:

Line 957:120: E501 line too long (138 > 119 characters)

Comment last updated at 2020-08-15 17:20:22 UTC

[mergify]

This pull request is now in conflict... :(

[mergify]

This pull request is now in conflict... :(

[asrafulashiq]

Hi, any progress on this issue?

[awaelchli]

After countless hours of trying to fix this problem, I have come to the realization that this "ddp" mode is inherently limited to a single trainer.fit or single trainer.test call. Due to the fact that the main process launches the program itself N-1 times for a single trainer.fit means that there is a conflict when we do subsequent calls

trainer.fit()
...
trianer.fit()
trainer.test()

(for example in a loop), or when we initialize Trainer multiple times in a script. In the example above, the first fit will launch the same script again in subprocesses. These train in parallel until fitting is completed and then they get killed. The main process will reach the second fit, and launch once again the same script in subprocesses again. Here is the problem. The subprocess now executes the first fit again because it simply executes the script from the beginning. There is no way we can reroute it to the right place, as no state is maintained across processes.

I will close this PR and propose to add a note or even warning to the docs that ddp backend has this limitation.

[williamFalcon]

we can maintain state using environment variables