pageserver: quieten some shutdown logs around logical size and flush

[jcsp]

Problem

Looking at several noisy shutdown logs:

• In test_timeline_deletion_with_files_stuck_in_upload_queue is flaky #7861 we're hitting a log error with InternalServerError(timeline shutting down\n' on the checkpoint API handler.
• In the field, we see initial_logical_size_calculation errors on shutdown, via DownloadError
• In the field, we see errors logged from layer download code (independent of the error propagated) during shutdown

Closes: #7861

Summary of changes

The theme of these changes is to avoid propagating anyhow::Errors for cases that aren't really unexpected error cases that we might want a stacktrace for, and avoid "Other" error variants unless we really do have unexpected error cases to propagate.

• On the flush_frozen_layers path, use the FlushLayerError type throughout, rather than munging it into an anyhow::Error. Give FlushLayerError an explicit from_anyhow helper that checks for timeline cancellation, and uses it to give a Cancelled error instead of an Other error when the timeline is shutting down.
• In logical size calculation, remove BackgroundCalculationError (this type was just a Cancelled variant and an Other variant), and instead use CalculateLogicalSizeError throughout. This can express a PageReconstructError, and has a From impl that translates cancel-like page reconstruct errors to Cancelled.
• Replace CalculateLogicalSizeError's Other(anyhow::Error) variant case with a Decode(DeserializeError) variant, as this was the only kind of error we actually used in the Other case.
• During layer download, drop out early if the timeline is shutting down, so that we don't do an error!() log of the shutdown error in this case.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3150 tests run: 3017 passed, 0 failed, 133 skipped (full report)

---

Flaky tests (1)

Postgres 15

• test_pageserver_restarts_under_worload: release

Code coverage* (full report)

• functions: 31.4% (6489 of 20673 functions)
• lines: 48.4% (50217 of 103749 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
361ae5b at 2024-05-30T09:26:43.284Z :recycle:}