[load_dataset] shard and parallelize the process

[stas00]

• Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
• If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:

• much faster build by parallelizing the build process
• if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq

[vanpersie32]

I need the same feature for distributed training

[lhoestq]

I think @TevenLeScao is exploring adding multiprocessing in GeneratorBasedBuilder._prepare_split - feel free to post updates here :)

[TevenLeScao]

Posted a PR to address the building side, still needs something to load sharded arrow files + tests

[mariosasko]

Closing as this feature has been implemented in #5107