Hardware specific parts of Accelerator Refactoring

[justusschock]

What does this PR do?

Adds the Hardware specific parts of the refactoring (#5616 ) as Accelerators for CPU, GPU and TPU, the accelerator connector and the specific plugins for single-device training, single tpu training and multiple tpu training (TPUSpawn).

Only files with new classes are added and some imports are changed. No integration into Trainer yet.

Should be merged after #5715 #5718 #5714

[pep8speaks]

Hello @justusschock! Thanks for updating this PR.

• In the file pytorch_lightning/accelerators/accelerator_connector.py:

Line 383:13: W503 line break before binary operator

Comment last updated at 2021-02-01 12:37:53 UTC

[codecov]

Codecov Report

Merging #5719 (ea88661) into release/1.2-dev (3bacac7) will decrease coverage by 4%.
The diff coverage is 30%.

@@               Coverage Diff                @@
##           release/1.2-dev   #5719    +/-   ##
================================================
- Coverage               89%     86%    -4%     
================================================
  Files                  173     181     +8     
  Lines                12495   13110   +615     
================================================
+ Hits                 11175   11226    +51     
- Misses                1320    1884   +564     

[Borda]

seems as last issue:
E ImportError: cannot import name 'ParallelLoader' from 'torch_xla' (/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch_xla/__init__.py)

[tchaton]

@justusschock Really good work there ! Not all comments should be adressed in this PR.

[tchaton]

I feel like __temp_weight_distributed_end.ckpt should be a GLOBAL property.

[justusschock]

what do you mean by global property?

[tchaton]

class

    TEMPORARY_WEIGHT_PATH = "__temp_weight_distributed_end.ckpt"
    def __init__

        
        ...

    def 

        path = os.path.join(original_model.trainer.default_root_dir, self.TEMPORARY_WEIGHT_PATH)

[tchaton]

Could we have a ColabEnv for TPU and move some logic there ? In another PR :)

[justusschock]

not sure I completely understand this... You mean a ClusterEnvironment for that? We can certainly look into that.

[tchaton]

Not sure Colab is a ClusterEnviromenent.

I meant more something like TPURunningEnviromenent -> Colab, Kaggle_colab, etc...

Best,
T.C