pageserver: use a single tokio runtime

[problame]

Before this PR, each core had 3 executor threads from 3 different runtimes. With this PR, we just have one runtime, with one thread per core. Switching to a single tokio runtime should reduce that effective over-commit of CPU and in theory help with tail latencies -- iff all tokio tasks are well-behaved and yield to the runtime regularly.

Are All Tasks Well-Behaved? Are We Ready?

Sadly there doesn't seem to be good out-of-the box tokio tooling to answer this question.

We believe all tasks are well behaved in today's code base, as of the switch to virtual_file_io_engine = "tokio-epoll-uring" in production (https://github.com/neondatabase/aws/pull/1121).

The only remaining executor-thread-blocking code is walredo and some filesystem namespace operations.

Filesystem namespace operations work is being tracked in #6663 and not considered likely to actually block at this time.

Regarding walredo, it currently does a blocking poll for read/write to the pipe file descriptors we use for IPC with the walredo process.
There is an ongoing experiment to make walredo async (#6628), but it needs more time because there are surprisingly tricky trade-offs that are articulated in that PR's description (which itself is still WIP). What's relevant for this PR is that

1. walredo is always CPU-bound
2. production tail latencies for walredo request-response (pageserver_wal_redo_seconds_bucket) are

• p90: with few exceptions, low hundreds of micro-seconds
• p95: except on very packed pageservers, below 1ms
• p99: all below 50ms, vast majority below 1ms
• p99.9: almost all around 50ms, rarely at >= 70ms
• Dashboard Link

The ones below 1ms are below our current threshold for when we start thinking about yielding to the executor.
The tens of milliseconds stalls aren't great, but, not least because of the implicit overcommit of CPU by the three runtimes, we can't be sure whether these tens of milliseconds are inherently necessary to do the walredo work or whether we could be faster if there was less contention for CPU.

On the first item (walredo being always CPU-bound work): it means that walredo processes will always compete with the executor threads.
We could yield, using async walredo, but then we hit the trade-offs explained in that PR.

tl;dr: the risk of stalling executor threads through blocking walredo seems low, and switching to one runtime cleans up one potential source for higher-than-necessary stall times (explained in the previous paragraphs).

Code Changes

• Remove the 3 different runtime definitions.
• Add a new definition called THE_RUNTIME.
• Use it in all places that previously used one of the 3 removed runtimes.
• Remove the argument from task_mgr.
• Fix failpoint usage where pausable_failpoint! should have been used. We encountered some actual failures because of this, e.g., hung get_metric() calls during test teardown that would client-timeout after 300s.

As indicated by the comment above THE_RUNTIME, we could take this clean-up further.
But before we create so much churn, let's first validate that there's no perf regression.

Performance

We will test this in staging using the various nightly benchmark runs.

However, the worst-case impact of this change is likely compaction (=>image layer creation) competing with compute requests.
Image layer creation work can't be easily generated & repeated quickly by pagebench.
So, we'll simply watch getpage & basebackup tail latencies in staging.

Additionally, I have done manual benchmarking using pagebench.
Report: https://neondatabase.notion.site/2024-03-23-oneruntime-change-benchmarking-22a399c411e24399a73311115fb703ec?pvs=4
Tail latencies and throughput are marginally better (no regression = good).
Except in a workload with 128 clients against one tenant.
There, the p99.9 and p99.99 getpage latency is about 2x worse (at slightly lower throughput).
A dip in throughput every 20s (compaction_period_ is clearly visible, and probably responsible for that worse tail latency.
This has potential to improve with async walredo, and is an edge case workload anyway.

Future Work

1. Once this change has shown satisfying results in production, change the codebase to use the ambient runtime instead of explicitly referencing THE_RUNTIME.
2. Have a mode where we run with a single-threaded runtime, so we uncover executor stalls more quickly.
3. Switch or write our own failpoints library that is async-native: async-native & less-global failpoints #7216

[github-actions]

2718 tests run: 2581 passed, 0 failed, 137 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_timeline_size_quota_on_startup: release

Code coverage* (full report)

• functions: 28.1% (6279 of 22338 functions)
• lines: 46.9% (44176 of 94105 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
a7fd969 at 2024-03-23T16:28:43.276Z :recycle:}

[problame]

Nightly pgbench runs in staging look unchanged 👍