Support for pipeline/model parallelism

[vfdev-5]

🚀 Feature

Following #1709, Ignite is not supporting other modern types of parallelism.

The idea is to introduce the following API to make ignite.distributed aware of data parallel group:

ranks = [0, 2]
# or
# dp_group = get_data_parallel_group()
idist.set_dp_group(ranks=ranks)

We think to provide basic examples of pipeline parallelism using DeepSpeed and FairScale with Ignite.

Additionally, helper methods (e.g. idist.all_reduce, etc) could have group as optional parameter.

cc @g-karthik

[Moh-Yakoub]

Hey @vfdev-5. Is this still WIP. are there any areas related to this task that I can participate in?

Thanks a lot

[vfdev-5]

@Moh-Yakoub thanks for asking. This is still WIP.

are there any areas related to this task that I can participate in?

This is a bit tricky and depends on your understanding how parallelism works.
Recently, I shared this very basic example to play around.
I think what would be of real help is, for example, fixing this issue: #1757
such that you can have an experience with ignite.distributed module. What do you think?

Meanwhile, you can also take a look at deepspeed and fairscale examples and try to code how to use ignite there... We can discuss about that later...

[Moh-Yakoub]

Great, I will take a look. I can start also trying to work with #1757