pageserver: fix a warning on secondary mode downloads after evictions

[jcsp]

Problem

In 4ce6e2d we added a warning when progress stats don't look right at the end of a secondary download pass.

This Correcting drift in progress stats warning fired in staging on a pageserver that had been doing some disk usage eviction.

The impact is low because in the same place we log the warning, we also fix up the progress values.

Summary of changes

• When we skip downloading a layer because it was recently evicted, update the progress stats to ensure they still reach a clean complete state at the end of a download pass.
• Also add a log for evicting secondary location layers, for symmetry with attached locations, so that we can clearly see when eviction has happened for a particular tenant's layers when investigating issues.

This is a point fix -- the code would also benefit from being refactored so that there is some "download result" type with a Skip variant, to ensure that we are updating the progress stats uniformly for those cases.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3132 tests run: 3005 passed, 0 failed, 127 skipped (full report)

---

Flaky tests (1)

Postgres 16

• test_pageserver_restarts_under_worload: release

Code coverage* (full report)

• functions: 31.4% (6449 of 20537 functions)
• lines: 48.3% (49935 of 103304 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
e32889f at 2024-05-24T15:31:43.639Z :recycle:}