tiered compaction: can't get last key

[arpad-m]

Found in #7283, the test_read_at_max_lsn test added by #7007 is failing with tiered compaction:

diff --git a/pageserver/src/tenant.rs b/pageserver/src/tenant.rs
index 0806ef0cf..70b6eaa44 100644
--- a/pageserver/src/tenant.rs
+++ b/pageserver/src/tenant.rs
@@ -3853,6 +3853,7 @@ mod tests {
     use bytes::BytesMut;
     use hex_literal::hex;
     use pageserver_api::keyspace::KeySpace;
+    use pageserver_api::models::CompactionAlgorithm;
     use rand::{thread_rng, Rng};
 
     static TEST_KEY: Lazy<Key> =
@@ -5151,7 +5152,9 @@ mod tests {
 
     #[tokio::test]
     async fn test_read_at_max_lsn() -> anyhow::Result<()> {
-        let harness = TenantHarness::create("test_read_at_max_lsn")?;
+        let mut harness = TenantHarness::create("test_read_at_max_lsn")?;
+        harness.tenant_conf.compaction_algorithm = CompactionAlgorithm::Tiered;
+
         let (tenant, ctx) = harness.load().await;
         let tline = tenant
             .create_test_timeline(TIMELINE_ID, Lsn(0x08), DEFAULT_PG_VERSION, &ctx)
@@ -5163,7 +5166,8 @@ mod tests {
         let test_key = Key::from_hex("010000000033333333444444445500000000").unwrap();
         let read_lsn = Lsn(u64::MAX - 1);
 
-        assert!(tline.get(test_key, read_lsn, &ctx).await.is_ok());
+        let result = tline.get(test_key, read_lsn, &ctx).await;
+        assert!(result.is_ok(), "result is not Ok: {}", result.unwrap_err());
 
         Ok(())
     }

originally part of #6768
moved to #7554

[arpad-m]

Preliminary results from investigation yesterday and today:

• the test first writes a bunch of keys, including 010000000033333333444444445500000000 (test_key), runs compaction, and then queries test_key, expecting to find it.
• legacy compaction finds the key, tiered compaction doesn't.
• for tiered compaction, it tries to look into an image layer that should contain the key but it doesn't
• I instrumented collect_keyspace to print the lsn and whether the returned range includes the key or not. It turns out that the returned range doesn't include the key! It is used both by new and old compaction to determine which keys to put into the image layer.
• However there is a difference between legacy and tiered compaction: collect_keyspace never gets called for any non-start lsns for legacy compaction. in other words, legacy compaction doesn't put test_key into an image but leaves it in a delta.
• tiered compaction creates an image however, which does call collect_keyspace for an lsn that's supposed to include the key. but the key never ends up in the image layer.

After a chat with @VladLazar things are more clearer now: the test writes the keys via TimelineWriter that bypasses DBDIR_KEY. So that's why they don't show up in collect_keyspace.

It seems to me that this is a faulty test issue rather than a problem with tiered compaction.

I'll try to make the legacy compaction issue an image layer to maybe reproduce it there as well.