all_gather raises NotImplementedError when no Accelerator defined in Trainer

[8greg8]

🐛 Bug

When no Accelerator is defined in Trainer, all_gather function in LightningModule raises NotImplementedError.

Please reproduce using the BoringModel and post here

https://colab.research.google.com/drive/1VPEIaQ-aN5KVA70VtvGk24AVkTYPvMoY?usp=sharing

To Reproduce

Expected behavior

• All gather should return a value instead.

Environment

• CUDA:
  • GPU:
    • Tesla P100-PCIE-16GB
  • available: True
  • version: 10.1
• Packages:
  • numpy: 1.19.4
  • pyTorch_debug: True
  • pyTorch_version: 1.7.0+cu101
  • pytorch-lightning: 1.1.2rc1
  • tqdm: 4.41.1
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor: x86_64
  • python: 3.6.9
  • version: Proposal for help #1 SMP Thu Jul 23 08:00:38 PDT 2020

Additional context

[awaelchli]

Fyi your colab notebook is not public, it can't be accessed :)

[8greg8]

@awaelchli sorry my bad. Corrected and changed the link in bug description. You should be able to access now.

[awaelchli]

@tchaton we should provide implementations for all_gather on cpu and single gpu to make code device agnostic. Does that make sense?

[tchaton]

Hey @awaelchli,

Yes, we should.

Best,
T.C

[tchaton]

Dear @8greg8,

I checked the notebook and couldn't reproduce the bug.

@awaelchli I have checked the code and it seems we are already supporting all_gather on cpu and single gpu.
However, we don't support gradients for TPU.

Best,
T.C

[tchaton]

Dear @8greg8,

I checked the notebook and couldn't reproduce the bug.

@awaelchli I have checked the code and it seems we are already supporting all_gather on cpu and single gpu.
However, we don't support gradients for TPU. I opened an issue for it.

Best,
T.C

[awaelchli]

Yes because that came automatically with the accelerator refactor and the discussion here is before that was introduced.
Today, Trainer always has an accelerator defined. This issue can be closed if you agree.