Work around flakyness in spill hysteresis

[crusaderky]

• Mitigates Memory may not shrink fast enough #5840
• Closes test_spill_hysteresis flaky on ubuntu #5848
• Partially reverts Allow memory monitor to evict data more aggressively #3424

[github-actions]

Unit Test Results

       12 files  ±0         12 suites  ±0   7h 14m 0s ⏱️ - 8m 4s
  2 622 tests +1    2 541 ✔️ +4    81 💤 ±0  0 ❌  - 3 
15 656 runs  +6  14 764 ✔️ +6  892 💤 +3  0 ❌  - 3 

Results for commit 8152265. ± Comparison against base commit 0e12374.

♻️ This comment has been updated with latest results.

[crusaderky]

Stress test (120 runs) is all green:
https://github.com/crusaderky/distributed/runs/5308278349?check_suite_focus=true

[fjetter]

I would like to start a discussion about how we write tests for memory related things.

• This particular test is very complex and slow
• We're allocating a lot of memory. I would prefer keeping unit tests low on resources even if 1GB should be not a problem
• Relying on OS RSS reporting appears to be fragile. We're still xfailing on OSX.

I believe this is a situation where I would prefer mocking. The one measurement we rely on that is critical and the culprit for all the flakiness is proc.memory_info().rss. This is a very stable API and we could easily mock this by encapsulating it in a function. In the end, what we want to test is our spill logic assuming psutil/OS does properly report RSS. We do not want to test how the OS manages memory, this is outside of our control.

[crusaderky]

Superseded by #5870