Create custom distributed plugin to allow model.parallelize

[SeanNaren]

🚀 Feature

Currently some models can be parallelized using the latest HF changes: huggingface/transformers#8696

We should create a plugin supported in this repo so that users can utilize this with the pl trainer.

[SeanNaren]

Update here. I think we're converging towards a hook that gives access to a distributed enabled environment. Before, setup used to be this hook since distributed communication was initialized, however after the accelerator refactor, things were standardized meaning the setup hook is now being called before distributed communication it initialized.

Discussion here: Lightning-AI/pytorch-lightning#6318

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.