[WIP] [doc] performance/scalability revamp

[stas00]

moved from #15213 so that we get the doc generator working

XXX: The previous PR has comments/suggestions that need to be integrated here

---

@lvwerra and I are working on a massive performance/scalability docs revamp:

So the rough plan is to make custom plans for each of the combinations [inference|training] * [1 gpu|many gpus|cpu] so that it's very easy for the user to follow the instructions that are specific to their needs.

So the proposed doc layout is:

• performance.mdx (main entry point)
• perf_infer.mdx
  • perf_infer_cpu.mdx
  • perf_infer_gpu_many.mdx
  • perf_infer_gpu_one.mdx
• perf_train.mdx
  • perf_train_gpu_many.mdx
  • perf_train_gpu_one.mdx
• scalability.mdx (rename from parallelism.mdx) (XXX: to do)

See the PR's changes for a rough layout of the content.

One big question is this: At the moment everything is pytorch-centric, as we don't have any info on tf/flax. Down the road we will either inject tf/flax-specific instructions into the current docs, or perhaps it'd be better to have dedicated docs for pt/tf/flax. It'd help a lot to decide ahead of time to avoid document renaming and potentially breaking links. If we plan to have these PT-specific perhaps let's embed _pt in the filenames?

@lvwerra

[HuggingFaceDocBuilder]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[lvwerra]

Hi @stas00, first stab at the single GPU section. The text is still WIP but you could have look at the sections and rough content to see if you agree.

Looking at the docs as a whole again here a few thoughts:

• Instead of dividing the whole section into speed and memory related techniques I thought an alternative could be to have a summary table at the beginning or end where the memory and speed impact of all methods is described. Could also help a user navigate the doc better.
• In the previous PR we discussed where to discuss common concepts (first time they occurred vs. in the performance docs). I am currently leaning towards the former. The main thing that's currently in the performance.mdx and I am not sure, yet, where I would put it is the hardware discussion which is quite different to everything else we are doing which makes me think that maybe we can create a dedicated document hardware.mdx where we could also expand a bit to other accelerators (TPU/IPUs for example).
• If we take that road the performance.mdx doc is again quite empty and I think we could move the content from perf_infer.mdx and perf_train.mdx to this document and then have a shared entry point and remove perf_infer.mdx and perf_train.mdx.

If you are happy with the structure I could start outlining the performance.mdx and polishing the single-GPU section while you could for example work on the multi-GPU training content.

What do you think?

[stas00]

Thank you for taking a stab at redesign, @lvwerra

It's very clear that we have fundamentally conflicting visions of how the performance documentation should be laid out. So it'd be wasteful to spend energy to try to come up with something that resonates with both of us.

I don't think this project can have 2 masterminds, and since I'm currently busy with BigScience training and we don't want this to fall between the cracks, I propose that if this works for you, please take over and proceed in an unimpeded way that sounds good to you, @lvwerra.

Once you have completed the restructuring and added what you know and you want me to fill in some gaps that I hopefully know to fill in please let me know and I will do so. or alternatively do tell me what sections are missing and I will work on those in free format and you can then adapt them to the structure you end up choosing.

[lvwerra]

Hi @LysandreJik and @sgugger,

I revamped the documentation a bit and reduced the scope from all the documents @stas00 lined out to just the performance.mdx and the training on single/multi-GPU so we can converge on those before adding the others, which also prevents a monster PR. For some reason the preview of the docs still does not work - if you have any idea, let me know.

To guide you a bit in the review:

• performance.mdx should be the entry point and give an overview
• perf_train_gpu_single.mdx shows all the tricks to efficiently train large models on a single GPU (gradient accumulation/checkpointing, optimizers, DeepSpeed Zero, data loaders etc.)
• perf_train_gpu_single.mdx shows all the tricks to efficiently train large models on a single GPU (mostly parallelism strategies: data, tensor, pipeline parallelism)
• perf_hardware.mdx is where I put some tips and tricks about custom hardware setups

In subsequent PRs we can:

• add more TensorFlow examples
• add TPU training guide
• add inference side
• add guides for train/inference on specialized hardware (Optimum)

This is the plan for the full documentation:

Let me know what you think about style and content of the documentation. There are still a few rough edges but I think it is good for a first review.

[sgugger]

The preview can't work if you don't rebase on master to have the new doc structure (everything nested in an "en" subfolder)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[lvwerra]

Thank you @sgugger it worked!

[sgugger]

This change does not seem to be linked to this PR.

[lvwerra]

Good catch!must have happened during rebase - I'll remove it.

[sgugger]

This is one year old, is it really the bleeding edge?

[stas00]

I did spend time investigating this a few months ago, you can see the details here: pytorch/pytorch#71274

the pytorch devs swear it's supposed to be faster, but I wasn't able to see any speed improvement. tldr: pytorch/pytorch#71274 (comment)

I have a separate PR that expands the optimizers section: #14708 and I was thinking to add it after the dust settles, but I suppose it might be better to put it into the "fodder" now?

[sgugger]

As long as it's not forgotten here I'm fine with both solutions.

[stas00]

@lvwerra, I rebased and solved the conflict introduced by #16860 - fixing it at its new file the content was moved to.

[lvwerra]

@stas00 thank you!

@sgugger @LysandreJik would you like to take a look at the PR? I integrated the previous suggestion.

[LysandreJik]

Impressive refactor!

[lvwerra]

@sgugger are you happy to merge this?

[sgugger]

Yes, LGTM :-)