feat(pageserver): use leases to temporarily block gc

[yliang412]

Part of #7497, extracts from #7996, closes #8063.

Problem

With the LSN lease API introduced in #7808, we want to implement the real lease logic to prevent GC from proceeding pass some LSN when garbage collecting layers in a timeline.

To do so, we keeps an additional in-memory mapping of LSNs to leases in GCInfo. This mapping is updated when

• a request is made to grant a new lease (insertion of new leases/renewal of existing leases)
• GCInfo is refreshed during GC (removal of expired leases)

We use this mapping during GC similar to how we use retain_lsns for branches. The idea is that we will keep a layer if the layer start LSN less or equal to any of the leased LSN. This guarantees that we will keep all the layers needed to reconstruct all pages at all the leased LSNs with valid leases at a given time.

Future Task

• For the current lease implementation, we are actually keeping all the layers below the maximum LSN with valid leases (same problem for retain_lsns). Theoretically, for each LSN with valid lease, we only need to keep down to the most recent image layer at that lease/branch LSN. This can be a win for reducing the amount of data we retain.

Summary of changes

• Maintain a in-memory mapping of LSN to lease expiration time in GCInfo.
• Requesting a lease for a LSN below latest_gc_cutoff_lsn will error out.
• Remove expired lease from the map when refreshing GCInfo.
• Block GC (similar to how we uses retain_lsns).
• A unit test test the make_lsn_lease API and how leases get used during GC.
• Delay GC by lease period at startup.
• Lease duration config in TenantConf (useful for running tests)

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3222 tests run: 3105 passed, 0 failed, 117 skipped (full report)

---

Code coverage* (full report)

• functions: 32.4% (6842 of 21098 functions)
• lines: 50.0% (53363 of 106812 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
cf488a8 at 2024-06-18T17:25:02.644Z :recycle:}