pageserver: clean up stray VM pages

[erikgrinaker]

In #9855 and #9914, we saw that ClearVmBits updates may be applied on shards that aren't responsible for VM pages. This can lead to stray writes to these keys, which can be incomplete since the shards will not have seen WAL records for these pages. We should clean these keys up, to avoid them cropping up again later and causing problems.

Bonus question: if we only apply ClearVmBits but not the page images on stray shards, why don't these error out during compaction when we presumably try to reconstruct the VM pages?

[erikgrinaker]

This is already done during compactions, which removes relation blocks that don't belong to the shard. This also explains why these stray keys didn't cause compaction failures.

neon/libs/pageserver_api/src/shard.rs

Lines 173 to 190 in a68edad

	/// Return true if the key should be discarded if found in this shard's
	/// data store, e.g. during compaction after a split.
	///
	/// Shards _may_ drop keys which return false here, but are not obliged to.
	pub fn is_key_disposable(&self, key: &Key) -> bool {
	if key_is_shard0(key) {
	// Q: Why can't we dispose of shard0 content if we're not shard 0?
	// A1: because the WAL ingestion logic currently ingests some shard 0
	// content on all shards, even though it's only read on shard 0. If we
	// dropped it, then subsequent WAL ingest to these keys would encounter
	// an error.
	// A2: because key_is_shard0 also covers relation size keys, which are written
	// on all shards even though they're only maintained accurately on shard 0.
	false
	} else {
	!self.is_key_local(key)
	}
	}

[erikgrinaker]

@jcsp indicated in #9786 (comment) that it's probably too expensive to force a recompaction of all tenants, since it requires downloading all layers from S3. So we'll just do this best-effort during compactions.

Let me know if I misinterpreted you @jcsp.

[jcsp]

Bonus question: if we only apply ClearVmBits but not the page images on stray shards, why don't these error out during compaction when we presumably try to reconstruct the VM pages?

Speaking from memory: the ShardedRange that we use to build the list of pages to include in compaction excludes them