Encapsulate spill buffer and memory_monitor

[fjetter]

A few mechanics in worker.py are very well self-contained, and it would make a lot of sense to encapsulate them into a separate module/class in order to reduce complexity:

• memory limit and thresholds
• initialisation of the SpillBuffer
• memory_monitor(), which handles pause, unpause, and force evict from spillbuffer

From the Worker's perspective, Worker.data should be treated as an opaque MutableMapping.

This is just a refactoring, with no functional impact.

Consider deprecating init arguments to Worker using target, spill, pause threshold etc. in favour of dask.config.

This change is part of the wider epic

• (Worker) State Machine determinism and replayability #5736

Related issues:

• Create MutableMapping for automatic compression #3656
• Asynchronous Disk Access in Workers #4424
• Thoughts on additional spilling layers #4629
• Change the memory-limit on the fly #5367
• Deserialise data on demand #5900

[gjoseph92]

From the Worker's perspective, Worker.data should be treated as an opaque MutableMapping.

This is just a refactoring, with no functional impact.

I see the value in doing this refactoring right now just to reduce code complexity. But as discussed in #4424 (comment), I think streaming data from disk to other workers will be a critical part of the new spilling design, and Worker.data will not be treatable as an opaque MutableMapping.

Spilling will need to be integrated into worker data transfer logic (and may even require changes to serialization and comms), which makes me wonder if spilling will actually make sense as a separate extension, or whether it needs to be a core part of the worker.

Or, maybe data transfer logic needs to move into some extensible WorkerDataStore interface? Either way, I think spilling and data transfer have to be somewhat coupled.

[fjetter]

think streaming data from disk to other workers will be a critical part of the new spilling design, and Worker.data will not be treatable as an opaque MutableMapping.

shuffling is out of scope for this. The shuffling plan right now also does not include the mutable mapping anyhow so I would suggest to postpone this until it becomes relevant again

[crusaderky]

As described in #5900, encapsulation is still highly desirable even if a MutableMapping is no longer fit for purpose, as the Worker needs to store/retrieve explicitly pickled or explicitly unpickled data.