page_service: rewrite batching to work without a timeout

[problame]

Problem

The timeout-based batching adds latency to unbatchable workloads.

We can choose a short batching timeout (e.g. 10us) but that requires high-resolution timers, which tokio doesn't have.
I thoroughly explored options to use OS timers (see this abandoned PR).
In short, it's not an attractive option because any timer implementation adds non-trivial overheads.

Solution

The insight is that, in the steady state of a batchable workload, the time we spend in get_vectored will be hundreds of microseconds anyway.

If we prepare the next batch concurrently to get_vectored, we will have a sizeable batch ready once get_vectored of the current batch is done and do not need an explicit timeout.

This can be reasonably described as pipelining of the protocol handler.

Implementation

We model the sub-protocol handler for pagestream requests (handle_pagrequests) as two futures that form a pipeline:

2. Batching: read requests from the connection and fill the current batch
3. Execution: take the current batch, execute it using get_vectored, and send the response.

The Reading and Batching stage are connected through a new type of channel called spsc_fold.

See the long comment in the handle_pagerequests_pipelined for details.

Changes

• Refactor handle_pagerequests
  • separate functions for
    • reading one protocol message; produces a BatchedFeMessage with just one page request in it
    • batching; tried to merge an incoming BatchedFeMessage into an existing BatchedFeMessage; returns None on success and returns back the incoming message in case merging isn't possible
    • execution of a batched message
  • unify the timeline handle acquisition & request span construction; it now happen in the function that reads the protocol message
• Implement serial and pipelined model
  • serial: what we had before any of the batching changes
    • read one protocol message
    • execute protocol messages
  • pipelined: the design described above
    • optionality for execution of the pipeline: either via concurrent futures vs tokio tasks
• Pageserver config
  • remove batching timeout field
  • add ability to configure pipelining mode
  • add ability to limit max batch size for pipelined configurations (required for the rollout, cf https://github.com/neondatabase/cloud/issues/20620 )
  • ability to configure execution mode
• Tests
  • remove batch_timeout parametrization
  • rename test_getpage_merge_smoke to test_throughput
    • add parametrization to test different max batch sizes and execution moes
  • rename test_timer_precision to test_latency
  • rename the test case file to test_page_service_batching.py
  • better descriptions of what the tests actually do

On the holding The TimelineHandle in the pending batch

While batching, we hold the TimelineHandle in the pending batch.
Therefore, the timeline will not finish shutting down while we're batching.

This is not a problem in practice because the concurrently ongoing get_vectored call will fail quickly with an error indicating that the timeline is shutting down.
This results in the Execution stage returning a QueryError::Shutdown, which causes the pipeline / entire page service connection to shut down.
This drops all references to the Arc<Mutex<Option<Box<BatchedFeMessage>>>> object, thereby dropping the contained TimelineHandles.

• => fixes page_service: figure out correctness of holding the TimelineHandle in the pending batch #9850

Performance

Local run of the benchmarks, results in this empty commit in the PR branch.

Key take-aways:

• concurrent-futures and tasks deliver identical batching_factor
  • tail latency impact unknown, cf page_service: measure tail latency impact in batchable workload #9837
• concurrent-futures has higher throughput than tasks in all workloads (=lower time metric)
• In unbatchable workloads, concurrent-futures has 5% higher CPU-per-throughput than that of tasks, and 15% higher than that of serial.
• In batchable-32 workload, concurrent-futures has 8% lower CPU-per-throughput than that of tasks (comparison to tput of serial is irrelevant)
• in unbatchable workloads, mean and tail latencies of concurrent-futures is practically identical to serial, whereas tasks adds 20-30us of overhead

Overall, concurrent-futures seems like a slightly more attractive choice.

Rollout

This change is disabled-by-default.

Rollout plan:

• https://github.com/neondatabase/cloud/issues/20620

Refs

• epic: Epic: get page throughput improvements #9376
• this sub-task: pageserver: batch get page requests and serve them with one vectored get #9377
• the abandoned attempt to improve batching timeout resolution: page_service: add benchmark for batching #9820
• closes page_service: figure out correctness of holding the TimelineHandle in the pending batch #9850
• fixes page_service: batching needless waits for unbatchable requests #9835

[VladLazar]

I think it's fine to merge given the cautious release plan. Nice job getting this into shape. This is tricky stuff!