Reapply "pageserver: revert flush backpressure" (#10270)

[erikgrinaker]

This reapplies #10135. Just removing this flush backpressure without further mitigations caused read amp increases during bulk ingestion (predictably), so it was reverted. We will replace it by compaction-based backpressure.

Problem

In #8550, we made the flush loop wait for uploads after every layer. This was to avoid unbounded buildup of uploads, and to reduce compaction debt. However, the approach has several problems:

• It prevents upload parallelism.
• It prevents flush and upload pipelining.
• It slows down ingestion even when there is no need to backpressure.
• It does not directly backpressure based on compaction debt and read amplification.

We will instead implement compaction-based backpressure in a PR immediately following this removal (#5415).

Touches #5415.
Touches #10095.

Summary of changes

Remove waiting on the upload queue in the flush loop.

[github-actions]

7359 tests run: 6973 passed, 1 failed, 385 skipped (full report)

---

Failures on Postgres 17

• test_scrubber_physical_gc_ancestors[None]: release-x86-64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_scrubber_physical_gc_ancestors[release-pg17-None]"

Flaky tests (3)

Postgres 17

• test_scrubber_physical_gc_ancestors[None]: debug-x86-64

Postgres 16

• test_pageserver_restart: release-arm64

Postgres 15

• test_metrics_normal_work: release-arm64

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
c7de608 at 2025-01-21T23:08:55.500Z :recycle:}