docs: rolling storage controller restarts RFC

[VladLazar]

Problem

Storage controller upgrades (restarts, more generally) can cause multi-second availability gaps.
While the storage controller does not sit on the main data path, it's generally not acceptable
to block management requests for extended periods of time (e.g. #8034).

Summary of changes

This RFC describes the issues around the current storage controller restart procedure
and describes an implementation which reduces downtime to a few milliseconds on the happy path.

Related #7797

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3806 tests run: 3700 passed, 0 failed, 106 skipped (full report)

---

Flaky tests (5)

Postgres 16

• test_lfc_resize: debug-x86-64
• test_storage_controller_node_deletion[True]: release-x86-64
• test_many_timelines: release-x86-64

Postgres 15

• test_scrubber_physical_gc_ancestors[2]: release-arm64

Postgres 14

• test_scrubber_physical_gc[None]: release-x86-64

Code coverage* (full report)

• functions: 32.3% (7306 of 22610 functions)
• lines: 50.4% (59080 of 117260 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
bfb0c59 at 2024-08-28T14:04:56.984Z :recycle:}

[jcsp]

Thinking of extra safety measures: we might in future like to carry an HTTP header on controller requests to pageservers, which would change for new leaders, so that pageservers could refuse requests from stale leaders. Might be worth embedding some counter in the leader table for that purpose.