Epic: revamp pageserver backpressure

[skyzh]

Followup on https://neondb.slack.com/archives/C03F5SM1N02/p1721058880447979 and #10095.

Updated proposal 2024-12-12 by @erikgrinaker:

Recall the current backpressure mechanism, based on these compute knobs:

• max_replication_write_lag: 500 MB (based on Pageserver last_received_lsn).
• max_replication_flush_lag: 10 GB (based on Pageserver disk_consistent_lsn).
• max_replication_apply_lag: disabled (based on Pageserver remote_consistent_lsn).

If the compute WAL leads by the given thresholds, the compute will inject a 10 ms sleep after every WAL record.

There are three aspects we don't backpressure on, but should:

• L0/compaction: if compaction falls behind, read amplification and compaction debt increases without bound (#5415).
• S3 uploads: if uploads fall behind, disk usage and crash recovery time increases without bound (#5897).
• Sharding: different shards can have different amounts of debt, so e.g. remote_consistent_lsn is misleading (#10095 comment).

With sharding, disk_consistent_lsn or remote_consistent_lsn are misleading, because they don't scale with shard count. 8 shards lagging by 1 GB LSN is very different from 1 shard lagging by 1 GB LSN -- we should bound the outstanding amount of work per shard, not the total outstanding work.

Additionally, the current backpressure protocol has a few issues:

• Calculation changes require compute release and restart (can take weeks/months).
• Protocol changes must be backwards compatible until all computes have restarted.
• Backpressure is binary (either off or 10 ms per WAL record).

Sketch for a new backpressure protocol:

• Each Pageserver shard computes a per-shard WAL target rate based on:
  • Safekeeper commit LSN: Pageserver ingestion should keep up.
  • In-memory layers: disk flushing should keep up.
  • L0 bytes and files: compaction should keep up.
  • Upload queue size: uploads should keep up.
• The Safekeeper aggregates a single WAL target rate based on min/average/sum across shards (needs experiments).
  • Alternatively, just have a stop or slow down signal from each shard.
• Send a single WAL target rate to the compute (0 to stall, -1 to disable throttling).
• Sleep on compute WAL appends based on target WAL rate.

Tasks

Tasks

Preview Give feedback

1. pageserver: backpressure on layer freeze/flush #7317
   a/tech_debt c/storage/pageserver +2
   
2. pageserver: add L0 compute backpressure (replace flush backpressure) #10095
   a/performance c/storage/pageserver +2
   
3. pageserver: single per-timeline variable backpressure signal #10116
   a/performance c/storage/pageserver +2
   
4. pageserver: WAL ingestion backpressure #5897
   c/storage/pageserver t/feature +2
5. pageserver: backpressure when delta layer count is high #5415
   c/storage/pageserver t/feature +2
   

[jcsp]

Let's look over our existing backpressure-related issues and make a plan

• pageserver: backpressure on layer freeze/flush  #7317
• pageserver: WAL ingestion backpressure #5897
• pageserver: backpressure when delta layer count is high #5415

[jcsp]

Plan:

• Prioritize: pageserver: backpressure on layer freeze/flush  #7317
• For the l0 stack problem: this may depend on other compaction design decisions? Potentially trigger compaction on LSN rather than time -- that way faster writing tenants get to compact more often.

Our existing mitigation for L0 compaction (only compact 10 at once) makes us safe.