pageserver: don't try to flush if shutdown during attach

[jcsp]

Problem

test_location_conf_churn fails on log errors when it tries to shutdown a pageserver immediately after starting a tenant attach, like this: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-8224/9761000525/index.html#/testresult/15fb6beca5c7327c

shutdown:shutdown{tenant_id=35f5c55eb34e7e5e12288c5d8ab8b909 shard_id=0000}:timeline_shutdown{timeline_id=30936747043353a98661735ad09cbbfe shutdown_mode=FreezeAndFlush}: failed to freeze and flush: cannot flush frozen layers when flush_loop is not running, state is Exited\n')

This is happening because Tenant::shutdown fires its cancellation token early if the tenant is not fully attached by the time shutdown is called, so the flush loop is shutdown by the time we try and flush.

Summary of changes

• In the early-cancellation case, also set the shutdown mode to Hard to skip trying to do a flush that will fail.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3006 tests run: 2891 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (1)

Postgres 15

• test_storage_controller_heartbeats[failure2]: release

Code coverage* (full report)

• functions: 32.7% (6936 of 21213 functions)
• lines: 50.0% (54323 of 108580 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
f2aceb8 at 2024-07-03T13:20:58.923Z :recycle:}