Threaded Dispatcher

[shikokuchuo]

Inviting @wlandau

Happy Friday again!

Just setting a marker here as I believe I have narrowed down a viable concept for threaded dispatcher.

Actually implementing it will be a fair amount of work, although this is now more firmly on the roadmap.

It will not affect current dispatcher - I am committed to keeping this as an option as I believe it is still a good idea. In particular I will prioritise the dispatcher extension interface which should allow crew to be fully asynchronous.

At least initially, it is likely not to be as full-featured as current dispatcher due to the limitations involved - in particular other threads will not be able to interact with R objects. Due to this major limitation, it is likely that even if this becomes the main way to send tasks, the separate dispatcher process is likely to be retained to run the crew extensions. I will be sure to reimplement this part so that crew extensions continue to work.

Work on this is likely to begin after integrating crew with dispatcher - I believe that is still the most valuable next step.

[shikokuchuo]

There is now a PR for this in #142. But the current version is limited and doesn't support crew yet.

[shikokuchuo]

That PR is now merged! I've opened #151 to carry on this thread.