release-7381

What's Changed

• Use updated patched azure SDK crates by @arpad-m in #10036
• Enable the pg_regress tests on staging for PG17 by @a-masterov in #9978
• fix(pageserver): fix gc-compaction racing with legacy gc by @skyzh in #10052
• safekeeper: use arc for global timelines and config by @oldmanfleming in #10051
• [proxy] impr: Additional logging for cancellation queries by @awarus in #10039
• Replace MAX_KEYS_PER_DELETE constant with function by @arpad-m in #10061
• test_runner: add visibility map test by @erikgrinaker in #9940
• Fix handling in-flight requersts in prefetch buffer resize by @knizhnik in #9968
• fix(pageserver): refresh_gc_info should always increase cutoff by @skyzh in #9862
• Print a log message if GetPage response takes too long by @hlinnaka in #10046
• feat(test_runner): allowed_errors in storage scrubber by @skyzh in #10062
• Emit nbtree vacuum cycle id in nbtree xlog through forced FPIs by @MMeent in #9932
• Disable readstream's reliance on seqscan readahead by @MMeent in #9860
• pageserver: fix ingest handling of CLog truncate by @jcsp in #10080
• CI(deploy): create git tags/releases before triggering deploy workflows by @jcgruenhage in #10022
• wal_decoder: fix compact key protobuf encoding by @VladLazar in #10074
• page_service: don't count time spent in Batcher towards smgr latency metrics by @problame in #10075
• pageserver: don't drop multixact slrus on non zero shards by @VladLazar in #10086
• compute_ctl: don't panic if control plane can't be reached by @myrrc in #10078
• Enable LFC for all the PG versions. by @a-masterov in #10068
• Increase neon_local http client to compute timeout in reconfigure. by @arssher in #10088
• walproposer: fix link error on debian 12 / ubuntu 22 by @myrrc in #10090
• Update compute_installed_extensions metric: by @lubennikovaav in #9891
• pageserver/storcon: add patch endpoints for tenant config metrics by @VladLazar in #10020
• storcon: allow for more concurrency in drain/fill operations by @VladLazar in #10093
• Do tenant manifest validation with index-part by @arpad-m in #10007
• test_runner: bump test_check_visibility_map timeout by @erikgrinaker in #10091
• Improve typing in test_runner/fixtures/httpserver.py by @tristan957 in #10103
• storcon: add safekeeper list API by @arpad-m in #10089
• Bump vm-builder v0.35.0 -> v0.37.1 by @mikhail-sakhnov in #10015
• Storage: create release PRs On Fridays by @bayandin in #10017
• tests: default interpreted proto in tests by @VladLazar in #10079
• Fix test_subscriber_synchronous_commit flakiness. by @arssher in #10057
• chore(proxy): remove code for old API by @conradludgate in #10109
• ci(fix): Use OIDC auth to login on ECR by @rahulinux in #10055
• Change the channel for notification. by @a-masterov in #10112
• Disable connection logging in pgbouncer by default by @hlinnaka in #10118
• fix(ci): Allow github-script to post test reports by @rahulinux in #10120
• storcon: refine logic for choosing AZ on tenant creation by @jcsp in #10054
• feat(pageserver): gc-compaction split over LSN by @skyzh in #9900
• use OIDC role instead of AWS access keys for managing test runner by @Bodobolero in #10117
• CI: don't run debug builds with LFC by @bayandin in #10123
• CI(deploy): fix git tag/release creation by @jcgruenhage in #10119
• Storage release 2024-12-13 by @vipvap in #10133

New Contributors

• @oldmanfleming made their first contribution in #10051
• @mikhail-sakhnov made their first contribution in #10015

Full Changelog: release-7336...release-7381

release-7336

What's Changed

• storage: cherry-pick SLRU, metrics and sharded ingest fixes into the release branch by @VladLazar in #10087

Full Changelog: release-7332...release-7336

release-7332

What's Changed

• pageserver: fix CLog truncate walingest by @skyzh in #10081

Full Changelog: release-7330...release-7332

release-7330

What's Changed

• Update AWS SDK crates by @arpad-m in #10056
• deps: update url to 2.5.4 for RUSTSEC-2024-0421 by @VladLazar in #10059
• Manual storage release 2024-12-09 by @VladLazar in #10060

Full Changelog: release-7327...release-7330

release-7327

What's Changed

• Update pgvector to 0.8.0 by @lubennikovaav in #9733
• test_runner: improve wait_until by @erikgrinaker in #9936
• storcon: add metric for AZ scheduling violations by @jcsp in #9949
• fix(proxy): forward notifications from authentication by @conradludgate in #9948
• CI(replication-tests): fix notifications about replication-tests failures by @bayandin in #9950
• proxy: Create Elasticache credentials provider lazily by @cloneable in #9967
• Update consensus protocol spec by @arssher in #9607
• Bump OTel, tracing, reqwest crates by @cloneable in #9970
• feat(proxy): emit JWT auth method and JWT issuer in parquet logs by @conradludgate in #9971
• Stop changing the value of neon.extension_server_port at runtime by @tristan957 in #9972
• fix(testing): Use 1 MB shared_buffers even with LFC by @ololobus in #9969
• storcon: use proper schedule context during node delete by @jcsp in #9958
• storcon_cli tenant-describe: include tenant-wide information in output by @problame in #9899
• page_service: batching observability & include throttled time in smgr metrics by @problame in #9870
• Add support for the extensions test for Postgres v17 by @a-masterov in #9748
• safekeeper,pageserver: add heap profiling by @erikgrinaker in #9778
• test_runner: use immediate shutdown in test_sharded_ingest by @erikgrinaker in #9984
• pageserver: only throttle pagestream requests & bring back throttling deduction for smgr latency metrics by @problame in #9962
• storcon: in shard splits, inherit parent's AZ by @jcsp in #9946
• pageserver: only store SLRUs & aux files on shard zero by @jcsp in #9786
• pageserver: more detailed logs when calling re-attach by @jcsp in #9996
• chore(proxy): remove postgres config parser and md5 support by @conradludgate in #9990
• pageserver: respond to multiple shutdown signals by @erikgrinaker in #9982
• Improvement: add console redirect timeout warning by @luixo in #9985
• chore(proxy): enforce single host+port by @conradludgate in #9995
• Support tenant manifests in the scrubber by @arpad-m in #9942
• tests & benchmarks: unify the way we customize the default tenant config by @problame in #9992
• test_page_service_batching: fix non-numeric metrics by @bayandin in #9998
• page_service: enable batching in Rust & Python Tests + Python benchmarks by @problame in #9993
• storcon: return an error for drain attempts while paused by @VladLazar in #9997
• optimize parms for ingest bench by @Bodobolero in #9999
• fix parsing human time output like "50m37s" by @Bodobolero in #10001
• pageserver: return proper status code for heatmap_upload errors by @erikgrinaker in #9991
• Assign /libs/proxy/ to proxy team by @cloneable in #10003
• feat(proxy): add option to forward startup params by @conradludgate in #9979
• feat(compute): Set default application_name for pgbouncer connections by @ololobus in #9973
• Display reqwest error source by @erikgrinaker in #10004
• Create a branch for compute release by @a-masterov in #9637
• tests: make storcon scale test AZ-aware by @jcsp in #9952
• pageserver: make BufferedWriter do double-buffering by @yliang412 in #9693
• chore: update clap by @conradludgate in #10009
• chore: update rust-postgres by @conradludgate in #10002
• pageserver: fix buffered-writer on macos build by @yliang412 in #10019
• proxy: Present new auth backend cplane_proxy_v1 by @awarus in #10012
• storage_controller: increase shard scan timeout by @erikgrinaker in #10000
• feat(compute): Allow specifying the reconfiguration concurrency by @ololobus in #10006
• Add compute_logical_snapshots_bytes metric by @tristan957 in #9887
• feat(pageserver): support schedule gc-compaction by @skyzh in #9809
• Bump anyhow to 1.0.94 by @tristan957 in #10028
• Send Content-Type header on reconfigure request from neon_local by @tristan957 in #10029
• Fix desc_str for Azure container by @arpad-m in #10021
• pageserver: only zero truncated FSM page on owning shard by @erikgrinaker in #10032
• pageserver: add wait_until_flushed parameter for timeline checkpoint by @erikgrinaker in #10013
• pageserver: add disk consistent and remote lsn metrics by @VladLazar in #10005
• pageserver: assert that keys belong to shard by @erikgrinaker in #9943
• chore: Reorganize and refresh CODEOWNERS by @ololobus in #10008
• pageserver: add metric for number of wal records received by each shard by @VladLazar in #10035
• test_runner/performance: add improved bulk insert benchmark by @erikgrinaker in #9812
• Bump sql_exporter to 0.16.0 by @tristan957 in #10041
• feat(pageserver): gc-compaction split job and partial scheduler by @skyzh in #9897
• Fix error message if PS connection is lost while receiving prefetch by @hlinnaka in #9923
• test(pageserver): disable gc_compaction smoke test for now by @skyzh in #10045
• page_service: don't count time spent flushing towards smgr latency metrics by @problame in #10042
• storcon: automatically clear Pause/Stop scheduling policies to enable detaches by @jcsp in #10011
• Storage release 2024-12-09 by @vipvap in #10053

New Contributors

• @luixo made their first contribution in #9985

Full Changelog: release-7261...release-7327

release-7261

What's Changed

• test_runner/performance: add logical message ingest benchmark by @erikgrinaker in #9749
• chore(proxy): vendor a subset of rust-postgres by @conradludgate in #9930
• storage controller: use proper ScheduleContext when evacuating a node by @jcsp in #9908
• safekeeper: use jemalloc by @erikgrinaker in #9780
• feat(compute_ctl): Always set application_name by @ololobus in #9934
• pageserver: download small objects using a smaller timeout by @jcsp in #9938
• Fixed fast_import pgbin in calling get_pg_version by @NanoBjorn in #9933
• Fix timeout value used in XLogWaitForReplayOf by @MMeent in #9937
• page_service: rewrite batching to work without a timeout by @problame in #9851
• Merge the consumption metric pushes by @cloneable in #9939
• Add GUC controlling whether to pause recovery if some critical GUCs at replica have smaller value than on primary by @knizhnik in #9057
• Compute image: prepare Postgres v14-v16 for Debian 12 by @bayandin in #9954
• Fix issues with prefetch ring buffer resize by @knizhnik in #9847
• storcon: don't take any Service locks in /status and /ready by @jcsp in #9944
• Storage & Compute release 2024-12-02 by @vipvap in #9959

Full Changelog: release-7245...release-7261

release-7245

What's Changed

• fix(pageserver): ensure all layers are flushed before measuring RSS by @skyzh in #9861
• Bump default Debian version to Bookworm everywhere by @bayandin in #9863
• Address beta clippy lint needless_lifetimes by @arpad-m in #9877
• Prefetch on macos by @knizhnik in #9875
• page_service: add benchmark for batching by @problame in #9820
• safekeeper: lift decoding and interpretation of WAL to the safekeeper by @VladLazar in #9746
• proxy: Demote all cplane error replies to info log level by @cloneable in #9880
• proxy: Demote errors from cplane request routines to debug by @cloneable in #9886
• fast_import: remove hardcoding of pg_version by @arpad-m in #9878
• chore(proxy): discard request context span during passthrough by @conradludgate in #9882
• track how much time the flush loop is stalled waiting for uploads by @Bodobolero in #9885
• Bump neon.logical_replication_max_snap_files default to 10000 by @tristan957 in #9896
• ingest benchmark: after effective_io_concurrency = 100 we can increase compute side parallelism by @Bodobolero in #9904
• CI(pre-merge-checks): build only one build-tools-image by @bayandin in #9718
• safekeeper: use protobuf for sending compressed records to pageserver by @VladLazar in #9821
• pageserver: add tenant config override for wal receiver proto by @VladLazar in #9888
• pageserver: add relsize cache metrics by @erikgrinaker in #9890
• fix(storage-scrubber): valid layermap error degrades to warning by @skyzh in #9902
• pageserver: add metrics for unknown ClearVmBits pages by @erikgrinaker in #9911
• proxy: promote two logs to error, fix multiline log by @cloneable in #9913
• fix(pageserver): flush deletion queue in reload shutdown mode by @skyzh in #9884
• pageserver: only apply ClearVmBits on relevant shards by @erikgrinaker in #9895
• proxy: spawn cancellation checks in the background by @awarus in #9918
• remote_storage/abs: count 404 and 304 for get as ok for metrics by @VladLazar in #9912
• Makefile: build pg_visibility by @erikgrinaker in #9922
• Update rust to 1.83.0, also update cargo adjacent tools by @arpad-m in #9926
• pageserver: return correct LSN for interpreted proto keep alive responses by @VladLazar in #9928
• test_runner: rerun all failed tests by @bayandin in #9917
• fix(compute_ctl): Allow usage of DB names with whitespaces by @ololobus in #9919
• Compute release 2024-11-28 by @ololobus in #9935

Full Changelog: release-7215...release-7245

release-7215

What's Changed

• pageserver: respect no_sync in VirtualFile by @jcsp in #9772
• Fix the regression test running against the staging instance by @a-masterov in #9773
• Use DATA_DIR instead of current workign directory in restore_from_wal script by @knizhnik in #9729
• adapt pgvector bench to minor version upgrades of PostgreSql by @Bodobolero in #9784
• tests: mitigate bug to stabilize test_storage_controller_many_tenants by @jcsp in #9771
• CI: remove separate check-build-tools-image workflow by @bayandin in #9708
• Bump default Postgres version for tests to v17 by @bayandin in #9777
• feat(compute/tests): Report successful replication test runs as well by @ololobus in #9787
• feat(pageserver): drop disposable keys during gc-compaction by @skyzh in #9765
• feat(page_service): timeout-based batching of requests by @VladLazar in #9321
• Remove at most one retain_lsn entry from (possibly offloaded) timelne's parent by @arpad-m in #9791
• Add the ability to configure GenericRemoteStorage for the scrubber by @arpad-m in #9652
• Move logic for ingest benchmark from GitHub workflow into python testcase by @Bodobolero in #9762
• Do not create DSM segment for wal_redo_postgres by @knizhnik in #9793
• chore(local_proxy): reduce some startup logging by @conradludgate in #9798
• fix(proxy/auth_broker): ignore -pooler suffix by @conradludgate in #9800
• PostgreSQL minor version updates (17.2, 16.6, 15.10, 14.15) by @hlinnaka in #9795
• re-acquire S3 OIDC token after long running tests for report upload to S3 by @Bodobolero in #9799
• build(deps): bump aiohttp from 3.10.2 to 3.10.11 by @dependabot in #9794
• fix(pageserver): queue stopped error should be ignored during create timeline by @skyzh in #9767
• scrubber: fix parsing issue with Azure by @arpad-m in #9797
• feat(pageserver): support key range for manual compaction trigger by @skyzh in #9723
• Register custom xlog reader callbacks for on-demand WAL download in StartupDecodingContext by @knizhnik in #9007
• CI(release): create reusable workflow for releases by @bayandin in #9806
• CODEOWNERS: add developer-productivity team by @bayandin in #9810
• compute_ctl: Streamline and Pipeline startup SQL by @MMeent in #9717
• Don't preload offloaded timelines by @arpad-m in #9646
• chore(proxy): demote a ton of logs for successful connection attempts by @conradludgate in #9803
• safekeeper: block deletion on protocol handler shutdown by @jcsp in #9364
• update macos version for runner by @fedordikarev in #9817
• CI(benchmarking): route test failures to on-call-qa-staging-stream by @bayandin in #9813
• scan_log_for_errors: check that regex is correct by @bayandin in #9815
• proxy: Rename RequestMonitoring to RequestContext by @cloneable in #9805
• tests: use fewer pageservers in test_sharding_split_smoke by @jcsp in #9804
• pageserver: avoid reading SLRU blocks for GC on shards >0 by @jcsp in #9423
• pageserver: enable compaction to proceed while live-migrating by @jcsp in #5397
• chore(proxy): demote more logs during successful connection attempts by @conradludgate in #9828
• pageserver: remove shard split hard link assertion by @VladLazar in #9829
• scrubber: allow restricting find_garbage to a partial tenant id prefix by @arpad-m in #9814
• [proxy] chore: allow bypassing empty params to /sql endpoint by @vadim2404 in #9827
• proxy: Refactor http conn pool by @awarus in #9785
• compute_ctl: print OpenTelemetry errors via tracing, not stdout by @arpad-m in #9830
• pageserver: revise metrics lifetime for SecondaryTenant by @jcsp in #9818
• Bump Postgres version by @knizhnik in #9808
• Python 3.11 by @bayandin in #9515
• fix(proxy): CancelKeyData display log masking by @conradludgate in #9838
• safekeeper,pageserver: add CPU profiling by @erikgrinaker in #9764
• Fix panic in compute_ctl metrics collection by @tristan957 in #9831
• minor PostgreSQL update in benchmarking by @Bodobolero in #9845
• Batch scrape workflows up to last 30 days and stop ad-hoc by @fedordikarev in #9846
• pageserver: permit reads behind GC cutoff during LSN grace period by @jcsp in #9833
• Avoid unnecessary send_replace calls in seqwait by @hlinnaka in #9852
• Write a newline after adding dynamic_shared_memory_type to PG conf by @tristan957 in #9843
• build(deps): bump mypy from 1.3.0 to 1.13.0 by @bayandin in #9670
• chore(proxy/jwks): reduce the rightward drift of jwks renewal by @conradludgate in #9853
• test_bulk_insert: fix typing for PgVersion by @bayandin in #9854
• proxy: Implement cancellation rate limiting by @awarus in #9739
• fix(pageserver): preempt and retry azure list operation by @skyzh in #9840
• safekeeper,pageserver: fix CPU profiling allowlists by @erikgrinaker in #9856
• test(pageserver): add detach ancestor smoke test by @skyzh in #9842
• fix(pageserver): ensure upload happens after delete by @skyzh in #9844
• Rename 'installed_extensions' metric to 'compute_installed_extensions' by @lubennikovaav in #9759
• fast imports: initial Importer and Storage changes by @problame in #9218
• test_runner: use LFC by default by @bayandin in #8613
• Bump futures-* crates, drop unused license, hide duplicate crate warnings by @cloneable in #9858
• Storage & Compute release 2024-11-25 by @vipvap in #9869
• chore: update rustls by @conradludgate in #9871
• Fixup Storage & Compute Release 2024-11-25 by @problame in #9881

Full Changelog: release-7145...release-7215

release-7145

What's Changed

• pageserver: refuse to load tenants with suspiciously old indices in old generations by @jcsp in #9719
• Improve error handling for NeonAPI fixture by @tristan957 in #9751
• safekeeper: fix atomicity of WAL truncation by @arssher in #9685
• cargo: use neon branch of rust-postgres by @VladLazar in #9757
• Correctly truncate VM by @knizhnik in #9342
• test: disable test_readonly_node_gc until proper fix by @yliang412 in #9755
• Remove the replication slot in test_snap_files at the end of the test by @tristan957 in #9752
• tests: add test_timeline_archival_chaos by @jcsp in #9609
• storcon: exclude non-Active tenants from shard autosplitting by @jcsp in #9743
• deny.toml: allow CDDL-1.0 license by @erikgrinaker in #9766
• tests: tolerate pageserver 500s in test_timeline_archival_chaos by @jcsp in #9769
• Correct mistakes in offloaded timeline retain_lsn management by @arpad-m in #9760
• Collect max_connections metric by @myrrc in #9770
• wal_decoder: make InterpretedWalRecord serde by @VladLazar in #9775
• Fix PG_MAJORVERSION_NUM typo by @tristan957 in #9774
• wal_decoder: rename end_lsn to next_record_lsn by @VladLazar in #9776
• safekeeper: send AppendResponse on segment flush by @erikgrinaker in #9692
• Cargo.toml: upgrade tikv-jemallocator to 0.6.0 by @erikgrinaker in #9779
• build(deps): bump smallvec to 1.13.2 to get UB fix by @problame in #9781
• Storage & Compute release 2024-11-18 by @vipvap in #9783

New Contributors

• @myrrc made their first contribution in #9770

Full Changelog: release-7125...release-7145

release-7125

What's Changed

• storcon: add peer token for peer to peer communication by @VladLazar in #9695
• CI: retry actions/github-script for 5XX errors by @bayandin in #9703
• fix(pageserver): increase frozen layer warning threshold; ignore in tests by @skyzh in #9705
• rfc: update aux file rfc to reflect latest optimizations by @skyzh in #9681
• safekeeper: add initialize_segment variant of safekeeper_wal_storage_operation_seconds by @erikgrinaker in #9691
• Remove checklist from the PR template by @bayandin in #9702
• new ingest benchmark by @Bodobolero in #9711
• test_runner: skip more tests using decorator instead of pytest.skip by @bayandin in #9704
• Clean up C.UTF-8 locale changes by @tristan957 in #9714
• feat(pageserver): support partial gc-compaction for delta layers by @skyzh in #9611
• use batch gh-workflow-stats-action with separate table by @fedordikarev in #9722
• Add a postgres_exporter config file by @tristan957 in #9720
• Add a retain_lsn test by @arpad-m in #9599
• Improve typing related to regress/test_logical_replication.py by @tristan957 in #9725
• Update pg_cron to 1.6.4 by @tristan957 in #9726
• safekeeper: don't flush control file on WAL ingest path by @erikgrinaker in #9698
• test(pageserver): add gc-compaction tests with delta will_init by @skyzh in #9724
• utils: remove unnecessary fsync in durable_rename() by @erikgrinaker in #9686
• Calculate compute_backpressure_throttling_seconds correctly by @tristan957 in #9732
• Add neon.logical_replication_max_logicalsnapdir_size by @tristan957 in #9467
• safekeeper: use WAL_SEGMENT_SIZE for empty timeline state by @erikgrinaker in #9734
• Fix autocommit footguns in performance tests by @tristan957 in #9735
• Do not wallog AUX files at replica by @knizhnik in #9457
• pageserver: add no_sync for use in regression tests (2/2) by @jcsp in #9678
• Add installed_extensions prometheus metric by @lubennikovaav in #9608
• tests: Increase timeout in test_create_churn_during_restart by @hlinnaka in #9736
• PostgreSQL minor version updates (17.1, 16.5, 15.9, 14.14) by @hlinnaka in #9727
• Compute release 2024-11-13 by @ololobus in #9745

Full Changelog: release-7097...release-7125