Fix init of WAL page header at startup

[hlinnaka]

If the primary is started at an LSN within the first of a 16 MB WAL segment, the "long XLOG page header" at the beginning of the segment was not initialized correctly. That has gone unnnoticed, because under normal circumstances, nothing looks at the page header. The WAL that is streamed to the safekeepers starts at the new record's LSN, not at the beginning of the page, so that bogus page header didn't propagate elsewhere, and a primary server doesn't normally read the WAL its written. Which is good because the contents of the page would be bogus anyway, as it wouldn't contain any of the records before the LSN where the new record is written.

Except that in the following cases a primary does read its own WAL:

1. When there are two-phase transactions in prepared state at checkpoint. The checkpointer reads the two-phase state from the XLOG_XACT_PREPARE record, and writes it to a file in pg_twophase/.

2. Logical decoding reads the WAL starting from the replication slot's restart LSN.

This PR fixes the problem with two-phase transactions. For that, it's sufficient to initialize the page header correctly. The checkpointer only needs to read XLOG_XACT_PREPARE records that were generated after the server startup, so it's still OK that older WAL is missing / bogus.

I have not investigated if we have a problem with logical decoding, however. Let's deal with that separately.

[hlinnaka]

I happened to notice this bug while I was writing a new test case for an issue with prepared transactions on the v17 branch. It turned out to be a pre-existing bug. (There were other issues with prepared transactions on the v17 branch too, though)

[MMeent]

@hlinnaka is the target branch the one you want this to be merged into? It seems like a fairly arbitrary branch to me.

[github-actions]

4977 tests run: 4813 passed, 0 failed, 164 skipped (full report)

---

Flaky tests (6)

Postgres 17

• test_scrubber_physical_gc[4]: release-arm64

Postgres 16

• test_slots_and_branching: release-x86-64

Postgres 15

• test_ondemand_wal_download_in_replication_slot_funcs: release-x86-64
• test_subscriber_restart: release-x86-64

Postgres 14

• test_scrubber_physical_gc[4]: release-arm64
• test_subscriber_restart: release-x86-64

Code coverage* (full report)

• functions: 31.9% (7435 of 23317 functions)
• lines: 49.9% (59921 of 120118 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
caea18d at 2024-09-20T23:57:30.168Z :recycle:}

[hlinnaka]

@hlinnaka is the target branch the one you want this to be merged into? It seems like a fairly arbitrary branch to me.

Right, it was just because I made those other change first and knew they were going to be merged soon. It's been merged now, so the point is moot.

[MMeent]

LGTM.

Please update the submodules after merging the PG PRs but before merging this one.

[clarkbw]

Leaving in reminder for this here: https://github.com/neondatabase/neon/pull/8916/files#diff-116a0da838509394db2ba0a7c6bec4c72808bb993040feba0bfde1e17fc4a93eR23

[hlinnaka]

Corresponding PRs in the postgres repo:

v17: neondatabase/postgres#506
v16: neondatabase/postgres#481
v15: neondatabase/postgres#482
v14: neondatabase/postgres#483

[hlinnaka]

Rebased, this is now ready to be merged again. I need approval for the v17 PR: neondatabase/postgres#506

[hlinnaka]

CI failure on one branch, seems to be flaky https://neon-github-public-dev.s3.amazonaws.com/reports/pr-8914/10957027921/index.html#suites/572721f46de364f1302c76c19e8023c4/c4814c8a998655dd/:

PG:2024-09-20 10:41:59.644 GMT [313604] LOG:  [WP] got VoteResponse from acceptor 127.0.0.1:18364, voteGiven=1, epoch=2, flushLsn=0/2000000, truncateLsn=0/14EE280, timelineStartLsn=0/14EE280
PG:2024-09-20 10:41:59.644 GMT [313604] LOG:  [WP] got votes from majority (1) of nodes, term 3, epochStartLsn 0/2000000, donor 127.0.0.1:18364, truncate_lsn 0/14EE280
2024-09-20T10:41:59.668726Z  INFO starting activity monitor for postgresql://cloud_admin@127.0.0.1:18368/postgres?application_name=compute_activity_monitor
PG:2024-09-20 10:41:59.644 GMT [313604] PANIC:  [WP] collected propEpochStartLsn 0/2000000, but basebackup LSN 0/14EE280
PG:2024-09-20 10:41:59.673 GMT [313607] LOG:  [NEON_SMGR] [shard 0] libpagestore: connected to 'postgresql://no_user@localhost:18361' with protocol version 2
PG:2024-09-20 10:41:59.753 GMT [313594] LOG:  background worker "WAL proposer" (PID 313604) was terminated by signal 6: Aborted

@arssher is this a known issue?

[hlinnaka]

CI failure on one branch, seems to be flaky https://neon-github-public-dev.s3.amazonaws.com/reports/pr-8914/10957027921/index.html#suites/572721f46de364f1302c76c19e8023c4/c4814c8a998655dd/:

PG:2024-09-20 10:41:59.644 GMT [313604] LOG:  [WP] got VoteResponse from acceptor 127.0.0.1:18364, voteGiven=1, epoch=2, flushLsn=0/2000000, truncateLsn=0/14EE280, timelineStartLsn=0/14EE280
PG:2024-09-20 10:41:59.644 GMT [313604] LOG:  [WP] got votes from majority (1) of nodes, term 3, epochStartLsn 0/2000000, donor 127.0.0.1:18364, truncate_lsn 0/14EE280
2024-09-20T10:41:59.668726Z  INFO starting activity monitor for postgresql://cloud_admin@127.0.0.1:18368/postgres?application_name=compute_activity_monitor
PG:2024-09-20 10:41:59.644 GMT [313604] PANIC:  [WP] collected propEpochStartLsn 0/2000000, but basebackup LSN 0/14EE280
PG:2024-09-20 10:41:59.673 GMT [313607] LOG:  [NEON_SMGR] [shard 0] libpagestore: connected to 'postgresql://no_user@localhost:18361' with protocol version 2
PG:2024-09-20 10:41:59.753 GMT [313594] LOG:  background worker "WAL proposer" (PID 313604) was terminated by signal 6: Aborted

@arssher is this a known issue?

The test case up to the failure is basically just:

1. Create branch, start an endpoint on it
2. SELECT pg_switch_wal()
3. Immediately stop the endpoint
4. Start a new endpoint

[arssher]

is this a known issue?

I haven't seen exactly this sequence with xlog switch, but it is kind of expected behaviour. CI failure misses first ep logs (it is better to do endpoint.stop(mode="immediate") to preserve them instead of endpoint.stop_and_destroy()), but I think the following happens:

1. SELECT pg_switch_wal() is done at first compute;
2. first compute is SIGKILL'ed
3. second compute starts and observers that sk position is 0/14EE280, nothing to sync, and fetches basebackup at this pos
4. safekeeper gets from TCP stream WAL switch record and bumps flush position to 0/2000000
5. basebackup is now spoiled because it is taken at wrong LSN position.

So I'd suggest to add sleep/retry until endpoint starts well. Alternatively safekeeper exposes list of walreceivers and we can poll sk until walreceiver on it is gone.

[hlinnaka]

is this a known issue?

I haven't seen exactly this sequence with xlog switch, but it is kind of expected behaviour.

I was able to reproduce this locally with:

    for i in range(1, 2000):
        branch_name = f"test_twophase{i}"
        env.neon_cli.create_branch(branch_name)
        endpoint = env.endpoints.create_start(branch_name)
        endpoint.safe_psql("SELECT pg_switch_wal()")
        endpoint.stop_and_destroy()
        endpoint = env.endpoints.create_start(branch_name)
        endpoint.safe_psql("SELECT pg_switch_wal()")
        endpoint.stop_and_destroy()

failed after about ~100 iterations

CI failure misses first ep logs (it is better to do endpoint.stop(mode="immediate") to preserve them instead of endpoint.stop_and_destroy()), but I think the following happens:

1. SELECT pg_switch_wal() is done at first compute;

2. first compute is SIGKILL'ed

3. second compute starts and observers that sk position is 0/14EE280, nothing to sync, and fetches basebackup at this pos

4. safekeeper gets from TCP stream WAL switch record and bumps flush position to 0/2000000

5. basebackup is now spoiled because it is taken at wrong LSN position.

So I'd suggest to add sleep/retry until endpoint starts well. Alternatively safekeeper exposes list of walreceivers and we can poll sk until walreceiver on it is gone.

Hmm, isn't this a potential problem in production too?

[hlinnaka]

Opened #9079 to track that flakiness, since it seems unrelated to this PR. In this PR, to work around this, I'll add a call to wait for the WAL to be flushed before stopping the compute.