pageserver: lsn lease logic needs to properly handle restart

[yliang412]

Problem

test_readonly_node_gc has become very flaky, and we root caused it to edge cases around pageserver restart / tenant migration. In particular, we will fail with Bad request: tried to request a page version that was garbage collected. if the GetPage request arrives before the first lease request after pageserver restart / tenant migration. The fix in #9055 does not eliminate bad requests.

To fix this permanently, we need to modify the lease logic to be aware of the what is guarded even after pageserver restart / tenant migration, likely through persistence.

Related: #8817, Notion

Potential Solution

Solution 1: Persisting state

• Persist lease information for each timeline in TimelineMetadata.
  • Persist max lsn with valid lease: not strictly necessary, but it is used in current GC logic for both retain lsns (branch points) and leases.
    • invariant: keep if layer.start_lsn < max_lsn_with_valid_lease (same for retain lsns)
  • Don't need timestamp, we can auto-renew the leases once we finishes pageserver restart / tenant migration
  • TODO: might need to persist all lsns, since there are gaps in between.
• Keep the existing logic: GC loop is only place we remove leases, lsn lease handler is the only place we add / renew leases.
  • TODO: synthetic size calculation handler currently also refreshes gc info? What should we do with those?

Solution 2: Prevent gc cutoff from proceeding

• Change the logic so that gc cutoff does not proceed pass lsn lease. (Pessimistic).
• Need to keep more data around.