Add distributed.metrics.monotonic

[crusaderky]

To be used in #6174

[github-actions]

Unit Test Results

       16 files  ±0         16 suites  ±0   7h 45m 45s ⏱️ + 16m 59s
  2 728 tests +1    2 645 ✔️ +1       80 💤  - 1  3 ❌ +1 
21 709 runs  +8  20 670 ✔️ +7  1 036 💤 ±0  3 ❌ +1 

For more details on these failures, see this check.

Results for commit fd31724. ± Comparison against base commit 370f456.

[mrocklin]

I think that maybe distributed.metrics.time would work on in that case? Maybe we can just use that.

[crusaderky]

I think that maybe distributed.metrics.time would work on in that case? Maybe we can just use that.

I'm worried about bad OS use cases. Specifically, the NTP sync could, for whatever reason, suddenly bring back the wall clock several seconds, instead of a few tens of milliseconds like it happens on a healthy box. If that ever happens, the spill system will send the worker in a complete apnoea until it's completely done, thus triggering timeouts.

An even more delicate use case is #6189. There, if the clock is sent back a few seconds during a particularly painful spill it may give enough time for the worker to rise in memory usage from 70% to 95% without having a chance to pause.