[FEA] Add experimental multi-threaded BypassMergeSortShuffleWriter #6060
Labels
feature request
New feature or request
P1
Nice to have for release
performance
A performance related task/issue
shuffle
things that impact the shuffle plugin
Milestone
This is a part of: #5039
This task is to add an experimental multi-threaded writer to handle compression and the IO required for writing shuffle blocks on the CPU. We have noticed that writing these blocks to temporary (map_id, reducer_id) files is CPU-bound in the lz4 case, and so adding more threads is beneficial for shuffle-intensive queries.
The text was updated successfully, but these errors were encountered: