[compute/postgres] feature: PostgreSQL 17 #8573

MMeent · 2024-07-31T18:02:47Z

Problem

We don't yet support PostgreSQL v17

Summary of changes

PG17 support, based on postgres/postgres@07b828e (RC1 / 2024-09-04)

Rebase our changes on top of v17
Get v17's patches into separated commits per target area
- Polish the commits to be actually separate: Several of them need some more work, and PG doens't compile for many of the intermediate commits.
Update extension code to compile against v17
Update Neon's code to compile against v17
Update & validate Pageserver's and Safekeeper's version-specific code for PG17
Update & validate pg_ffi/pgxn code
~~[ ] Patch PG17's ReadStream API internals to always prefetch calls, so including for sequential reads~~ To be implemented in a follow-up
~~[ ] Patch Neon's WALReader to pull WAL-pages from buffers (where possible) in PG17.~~ To be implemented as a follow-up
~~[ ] Add all supported extension~~ To be implemented as a follow-up
Rebase on top of 17beta3 and/or 17rc1 if/when they release before this PR is merged

NOTICE: The data produced by this version of the PostgreSQL fork WILL NOT BE
COMPATIBLE with the future full release of PostgreSQL 17 due to expected
future changes in magic numbers.
DO NOT EXPECT DATA IN V17-TENANTS TO BE COMPATIBLE WITH THE 17.0 RELEASE!

Part of https://github.com/neondatabase/cloud/issues/12088

Checklist before requesting a review

I have performed a self-review of my code.
If it is a core feature, I have added thorough tests.
Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

Do not forget to reformat commit message to not include the above checklist

github-actions · 2024-07-31T18:15:03Z

4968 tests run: 4804 passed, 0 failed, 164 skipped (full report)

Flaky tests (3)

Postgres 17

test_pageserver_compaction_smoke: release-arm64
test_scrubber_physical_gc[4]: debug-x86-64

Postgres 16

test_obsolete_slot_drop[cross-validation]: release-arm64

Code coverage* (full report)

functions: 31.8% (7412 of 23294 functions)
lines: 49.9% (59562 of 119380 lines)

* collected from Rust tests only

_{The comment gets automatically updated with the latest test results
630780d at 2024-09-12T17:21:43.588Z :recycle:}

TODO: - Updates & Validation of Pageserver, Safekeeper code with PG17 - Updates & Validation of pg_ffi/pgxn code - Patch PG17's ReadStream API to prefetch sequential reads - Rebase on top of Beta3 if/when that releases NOTICE: The data produced by this version of the PostgreSQL fork WILL NOT BE COMPATIBLE with the future full release of PostgreSQL 17 due to expected future changes in magic numbers. DO NOT EXPECT THIS DATA TO SURVIVE THE 17.0 RELEASE!

TODO figure out why this change was needed, run other tests.

…ress. TODO: run other tests, code cleanup

- PG17 is now up-to-date up to 2366ab24, which is >beta3 - PG17 now includes the patches up to v16's 47a9122 ("Preserve combocid v16") - Various operations in the write path have been vector-ified. Future work will go into vectorizing our read- and prefetch paths. - Rebased the patches on top of neon/HEAD @ b83d722 - Compiling Neon extensions now only requires the postgres headers This saves time by not having to compile PostgreSQL again every time we want to recompile the extension. We compile again, but tests TBD.

This reverts commit ad59042.

test_runner/regress/test_twophase.py

hlinnaka · 2024-09-11T18:45:05Z

It was useful whille developing the test. It could be removed now, although it doesn't do any harm either

…ot at a page header

.github/workflows/_build-and-test-locally.yml

test_runner/regress/test_download_extensions.py

test_runner/regress/test_compatibility.py

test_runner/regress/test_postgres_version.py

test_runner/fixtures/neon_fixtures.py

pageserver/src/pgdatadir_mapping.rs

jcsp

Some comments, but nothing that can't be picked up in a followup PR.

MMeent force-pushed the feat/pg-17-preps branch from 2ccea7f to 4013caa Compare August 2, 2024 10:48

lubennikovaav force-pushed the feat/pg-17-preps branch 2 times, most recently from 8e2807e to 77c2750 Compare August 13, 2024 12:58

ololobus assigned MMeent and lubennikovaav Aug 20, 2024

MMeent force-pushed the feat/pg-17-preps branch from f221f77 to 85732d2 Compare August 21, 2024 16:50

bayandin added the run-extra-build-macos When placed on a PR, tells the CI to run a build on macOS. No unit tests are run, though. label Aug 22, 2024

MMeent and others added 22 commits August 23, 2024 18:27

Add pg17 git submodule

35fadc1

Fix compilation issues in extensions when compiling against PG17

cdd2e6e

Further compilation fixes, and add CI steps for PG17

e45bf51

Make dockerfiles compatible with PG17

496ce3b

Disable all extensions in Dockerfile for PG17

48c2a5f

run debug tests on PG17

96ff9f4

add 17 as supported version

8160448

WIP add more v17 support. TODO: add switches

fbea1e8

add pagestore_smgr.c. This version passes 1 python test

24bac62

Fix Makefile

156cecd

DRAFT changes to rust code. don't work now

c4eff7e

WIP fix wal-redo. Passes test_normal_work

2378e76

WIP Fix inmem-smgr. Most of the tests from test_pg_regress pass.

6d4c012

TODO figure out why this change was needed, run other tests.

WIP fix pagestore_smgr. This version passes all test from test_pg_reg…

90f99f3

…ress. TODO: run other tests, code cleanup

Update Dockerfile to use v17. TODO: pass version from CI job?

065dc86

Use v17 in build_and_test.yml. TODO: code cleanup, add version choice

5493372

fix python codestyle

8cb1fa3

Use v16 in build_and_test.yml for neon-test-extensions steps

20f4e29

cargo fmt fix

1a9b2f1

Use v17 in Makefile and CI. TODO: code cleanup, add version choice

77110a9

MMeent added 6 commits September 10, 2024 01:24

ruff fixup

bf1daaa

Fixup! Bad merge for test_twophase.py

5199900

Fixup! Ruff formatting

a452b2c

Add check for alignedness of timeline writes

ad59042

Revert "Add check for alignedness of timeline writes"

22cc514

This reverts commit ad59042.

Add check for valid record LSNs

42c4348

MMeent force-pushed the feat/pg-17-preps branch from 5d7e4a0 to 42c4348 Compare September 10, 2024 15:54

MMeent added 2 commits September 11, 2024 16:54

ruff & fix assertion

4356d66

Reset poetry.lock to main branch's, and fix Lsn description.

3cdcee0

MMeent marked this pull request as ready for review September 11, 2024 16:34

MMeent requested review from a team as code owners September 11, 2024 16:34

MMeent requested review from yliang412 and NanoBjorn September 11, 2024 16:34

ololobus approved these changes Sep 11, 2024

View reviewed changes

test_runner/regress/test_twophase.py Outdated Show resolved Hide resolved

MMeent added 2 commits September 11, 2024 23:34

Revert assertion in Lsn constructor, and make sure branchpoints are n…

8d1dcd3

…ot at a page header

Fixup! lsn % op and reduced logging verbosity

6d6a792

NanoBjorn reviewed Sep 12, 2024

View reviewed changes

.github/workflows/_build-and-test-locally.yml Outdated Show resolved Hide resolved

Clean up de-cached build steps

630780d