storage controller: use proper ScheduleContext when evacuating a node

[jcsp]

Problem

When picking locations for a shard, we should use a ScheduleContext that includes all the other shards in the tenant, so that we apply proper anti-affinity between shards. If we don't do this, then it can lead to unstable scheduling, where we place a shard somewhere that the optimizer will then immediately move it away from.

We didn't always do this, because it was a bit awkward to accumulate the context for a tenant rather than just walking tenants.

This was a TODO in handle_node_availability_transition:

                        // TODO: populate a ScheduleContext including all shards in the same tenant_id (only matters
                        // for tenants without secondary locations: if they have a secondary location, then this
                        // schedule() call is just promoting an existing secondary)

This is a precursor to #8264, where the current imperfect scheduling during node evacuation hampers testing.

Summary of changes

• Add an iterator type that yields each shard along with a schedulecontext that includes all the other shards from the same tenant
• Use the iterator to replace hand-crafted logic in optimize_all_plan (functionally identical)
• Use the iterator in handle_node_availability_transition to apply proper anti-affinity during node evacuation.

[VladLazar]

Nice!

[github-actions]

6967 tests run: 6659 passed, 0 failed, 308 skipped (full report)

---

Flaky tests (4)

Postgres 17

• test_pageserver_compaction_circuit_breaker: debug-x86-64
• test_metric_collection_cleans_up_tempfile: debug-x86-64
• test_location_conf_churn[3]: debug-x86-64
• test_scrubber_physical_gc_ancestors[2]: debug-x86-64

Code coverage* (full report)

• functions: 30.6% (7988 of 26068 functions)
• lines: 48.6% (63486 of 130635 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
252d6b5 at 2024-11-29T13:40:59.579Z :recycle:}