pageserver: refuse to load tenants with suspiciously old indices in old generations

[jcsp]

Problem

Historically, if a control component passed a pageserver "generation: 1" this could be a quick way to corrupt a tenant by loading a historic index.

Follows #9383
Closes #6951

Summary of changes

• Introduce a Fatal variant to DownloadError, to enable index downloads to signal when they have encountered a scary enough situation that we shouldn't proceed to load the tenant.
• Handle this variant by putting the tenant into a broken state (no matter which timeline within the tenant reported it)
• Add a test for this case

In the event that this behavior fires when we don't want it to, we have ways to intervene:

• "Touch" an affected index to update its mtime (download+upload S3 object)
• If this behavior is triggered, it indicates we're attaching in some old generation, so we should be able to fix that by manually bumping generation numbers in the storage controller database (this should never happen, but it's an option if it does)

[github-actions]

5400 tests run: 5173 passed, 0 failed, 227 skipped (full report)

---

Flaky tests (1)

Postgres 16

• test_create_churn_during_restart: release-x86-64

Code coverage* (full report)

• functions: 31.8% (7889 of 24837 functions)
• lines: 49.4% (62455 of 126301 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
e33af1b at 2024-11-13T17:21:51.883Z :recycle:}