Fix distributed.worker.memory.target=False

[crusaderky]

Fix bug with config

distributed:
  worker:
    memory:
      target: false
      spill: 0.7

Where the Worker would ignore the target and use 0.7 instead.

Review a few tests around spilling.

xrefs

• Change the memory-limit on the fly #5367
• Constrained spill  #5543
• Fix flaky test_robust_to_bad_sizeof_estimates #5753

[crusaderky]

This would hit the spill threshold after 1200/0.6*0.7=1400 bytes worth of total process memory, and the pause threshold after 1200/0.6*0.8=1600 bytes. Unmanaged memory alone is always > 100 MB.

In other words, any futures at all managed to complete just thanks to a race condition with memory_monitor().
It was also a race condition to decide what would spill any of the keys, the zict LRU machinery used by target or the explicit evict() used by by spill.

Test reimplemented below.

[fjetter]

I see the test test_gather_then_submit_after_failed_workers fail quite deterministically. It looks like another instance of #5751

[github-actions]

Unit Test Results

       17 files  ±  0         17 suites  ±0   9h 7m 29s ⏱️ + 19m 40s
  2 595 tests ±  0    2 513 ✔️ ±  0       79 💤  -   1  3 ❌ +1 
21 828 runs   - 27  20 447 ✔️  - 86  1 374 💤 +56  7 ❌ +3 

For more details on these failures, see this check.

Results for commit b36fcf0. ± Comparison against base commit 8a5e014.