use Dask to parallelize document learning

[dlqqq]

Description

Uses Dask to parallelize document learning via /learn. The current approach sees roughly a 2-3x performance boost.

Before:

(LearnActor pid=13273) [/learn] Finished chunking documents. Time: 1308ms
(LearnActor pid=13273) [/learn] Completed. Time: 8117ms

After:

(LearnActor pid=33141) [/learn] Finished chunking documents. Time: 2685ms
(LearnActor pid=33141) [/learn] Finished computing embeddings. Time: 4571ms
(LearnActor pid=33141) [/learn] Complete. Time: 4573ms

Notes

The majority of the time spent by Dask is actually from the overhead of spawning a process. The grey bars represent deserialization of pickled data in each worker process:

There are some substantial improvements that can be made here.

[3coins]

@dlqqq
Thanks for putting this together. This looks great. Can you explore how the actual request/reply will work with dask setup.

[dlqqq]

Superseded by #244.