Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handmade Release 2024-05-13 #7735

Merged
merged 58 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
e6da7e2
Add option allowing running multiple endpoints on the same branch.
arssher May 2, 2024
0115fe6
Make 'neon.protocol_version = 2' the default (#7616)
hlinnaka May 6, 2024
3764dd2
pageserver: call maybe_freeze_ephemeral_layer from a dedicated task (…
jcsp May 6, 2024
67a2215
pageserver: label tenant_slots metric by slot type (#7603)
jcsp May 6, 2024
69337be
Fix grammar in provider.rs error message
tristan957 May 6, 2024
df1def7
refactor(pageserver): remove --update-init flag (#7612)
problame May 6, 2024
a96e15c
test: less flaky test_synthetic_size_while_deleting (#7622)
koivunej May 6, 2024
f3af5f4
Fix test_ts_of_lsn_api flakiness (#7599)
arpad-m May 6, 2024
7dd58e1
On-demand WAL download for walsender (#6872)
May 6, 2024
f1b654b
proxy: reduce number of concurrent connections (#7620)
khanova May 6, 2024
ac7dc82
use less `neon_local --pageserver-config-override` / `pageserver -c` …
problame May 6, 2024
af849a1
pageserver: post-shard-split layer trimming (1/2) (#7572)
jcsp May 7, 2024
3c9b484
feat: Timeline detach ancestor (#7456)
koivunej May 7, 2024
6e4e578
build(deps): bump werkzeug from 3.0.1 to 3.0.3 (#7625)
dependabot[bot] May 7, 2024
5a3d8e7
build(deps): bump jinja2 from 3.1.3 to 3.1.4 (#7626)
dependabot[bot] May 7, 2024
51376ef
Add Postgres commit sha to Postgres version (#4603)
bayandin May 7, 2024
2dbd1c1
build(deps): bump flask-cors from 3.0.10 to 4.0.1 (#7633)
dependabot[bot] May 7, 2024
ea531d4
fix(test suite): forward compat test is not using latest neon_local (…
problame May 7, 2024
d041f9a
refactor(rtc): remove excess cloning (#7635)
koivunej May 7, 2024
308227f
remove `neon_local --pageserver-config-override` (#7614)
problame May 7, 2024
017c34b
feat(pageserver): generate basebackup from aux file v2 storage (#7517)
skyzh May 7, 2024
0af66a6
pageserver: include generation number in local layer paths (#7609)
jcsp May 7, 2024
0c99e5e
proxy: cull http connections (#7632)
conradludgate May 7, 2024
b158a5e
compute_ctl: Non-functional prep changes to reduce diff (#7577)
sharnoff May 1, 2024
d709bcb
compute_ctl: Break up main() into discrete phases (#7577)
sharnoff May 1, 2024
26b1483
compute_ctl: Lift drop(startup_context_guard) into main() (#7577)
sharnoff May 4, 2024
b827e7b
compute_ctl: Fix unused variable on non-Linux (#7646)
sharnoff May 7, 2024
586e77b
tests: common log allow list for ancestor detach tests (#7645)
jcsp May 8, 2024
02d4286
`neon_local init`: write `pageserver.toml` directly; no `pageserver -…
problame May 8, 2024
e3a2631
proxy: do not invalidate cache for permit errors (#7652)
conradludgate May 8, 2024
b6d547c
Tiered compaction: add order asserts after delta key k-merge (#7648)
arpad-m May 8, 2024
870786b
Improve tiered compaction tests (#7643)
arpad-m May 8, 2024
a4a4d78
build(deps): bump moto from 4.1.2 to 5.0.6 (#7653)
bayandin May 8, 2024
8728d5a
neon_local: use `pageserver.toml` as source of truth for `struct Page…
problame May 8, 2024
0457980
Fix flaky test_gc_of_remote_layers (#7647)
petuhovskiy May 8, 2024
21e1a49
Expose LSN and replication delay as metrics (#7610)
May 8, 2024
1173ee6
chore(neon_test_utils): restrict installation to superuser (#7624)
skyzh May 8, 2024
ca154d9
pageserver: local layer path followups (#7640)
jcsp May 8, 2024
b06eec4
Ignore page header when comparing VM pages in test_vm_bits.py (#7499)
knizhnik May 8, 2024
d5399b7
pageserver: fix division by zero in layer counting metric (#7662)
VladLazar May 8, 2024
ab10523
remote_storage: AWS_PROFILE with endpoint overrides in ~/.aws/config …
problame May 9, 2024
39c712f
tests: adjust log allow list since reqwest upgrade (#7666)
jcsp May 9, 2024
107f535
storage controller: fix handing of tenants with no timelines during s…
jcsp May 9, 2024
41fb838
Fix tiered compaction k-merge bug and use in-memory alternative (#7661)
arpad-m May 9, 2024
2682e02
Revert "chore(neon_test_utils): restrict installation to superuser" (…
skyzh May 9, 2024
5ea117c
build(deps): bump Npgsql from 8.0.2 to 8.0.3 in /test_runner/pg_clien…
dependabot[bot] May 9, 2024
b9fd8dc
fix(test): update the config for neon_binpath in from_repo_dir (#7684)
skyzh May 9, 2024
be1a88e
Proxy added per ep rate limiter (#7636)
khanova May 10, 2024
13d9589
pageserver: don't call get_vectored with empty keyspace (#7686)
jcsp May 10, 2024
873b222
use own arm64 gha runners (#7373)
cicdteam May 10, 2024
0b02043
Fix permissions for safekeeper failpoints (#7669)
petuhovskiy May 10, 2024
86905c1
openapi: resolve the synthetic_size duplication (#7651)
koivunej May 10, 2024
d7f34bc
draw_timeline_dir: draw branch points and gc cutoff lines (#7657)
koivunej May 10, 2024
6206f76
build: run doctests (#7697)
koivunej May 10, 2024
d7c68dc
Tiered compaction: fix early exit check in main loop (#7702)
arpad-m May 10, 2024
95098c3
Fix checkpoint metric (#7701)
lubennikovaav May 10, 2024
6351313
feat: allow detaching from ancestor for timelines without writes (#7639)
koivunej May 10, 2024
1010431
Revert protocol version upgrade (#7727)
VladLazar May 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .config/nextest.toml
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
[profile.default]
slow-timeout = { period = "20s", terminate-after = 3 }
slow-timeout = { period = "60s", terminate-after = 3 }
4 changes: 1 addition & 3 deletions .github/actionlint.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
self-hosted-runner:
labels:
- arm64
- dev
- gen3
- large
# Remove `macos-14` from the list after https://github.com/rhysd/actionlint/pull/392 is merged.
- macos-14
- large-arm64
- small
- us-east-2
config-variables:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-build-tools-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
matrix:
arch: [ x64, arm64 ]

runs-on: ${{ fromJson(format('["self-hosted", "dev", "{0}"]', matrix.arch)) }}
runs-on: ${{ fromJson(format('["self-hosted", "gen3", "{0}"]', matrix.arch == 'arm64' && 'large-arm64' || 'large')) }}

env:
IMAGE_TAG: ${{ inputs.image-tag }}
Expand Down
24 changes: 3 additions & 21 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -236,27 +236,6 @@ jobs:
submodules: true
fetch-depth: 1

- name: Check Postgres submodules revision
shell: bash -euo pipefail {0}
run: |
# This is a temporary solution to ensure that the Postgres submodules revision is correct (i.e. the updated intentionally).
# Eventually it will be replaced by a regression test https://github.com/neondatabase/neon/pull/4603

FAILED=false
for postgres in postgres-v14 postgres-v15 postgres-v16; do
expected=$(cat vendor/revisions.json | jq --raw-output '."'"${postgres}"'"')
actual=$(git rev-parse "HEAD:vendor/${postgres}")
if [ "${expected}" != "${actual}" ]; then
echo >&2 "Expected ${postgres} rev to be at '${expected}', but it is at '${actual}'"
FAILED=true
fi
done

if [ "${FAILED}" = "true" ]; then
echo >&2 "Please update vendor/revisions.json if these changes are intentional"
exit 1
fi

- name: Set pg 14 revision for caching
id: pg_v14_rev
run: echo pg_rev=$(git rev-parse HEAD:vendor/postgres-v14) >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -362,6 +341,9 @@ jobs:
env:
NEXTEST_RETRIES: 3
run: |
#nextest does not yet support running doctests
cargo test --doc $CARGO_FLAGS $CARGO_FEATURES

for io_engine in std-fs tokio-epoll-uring ; do
NEON_PAGESERVER_UNIT_TEST_VIRTUAL_FILE_IOENGINE=$io_engine ${cov_prefix} cargo nextest run $CARGO_FLAGS $CARGO_FEATURES
done
Expand Down
33 changes: 21 additions & 12 deletions .github/workflows/neon_extra_builds.yml
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ jobs:
check-linux-arm-build:
needs: [ check-permissions, build-build-tools-image ]
timeout-minutes: 90
runs-on: [ self-hosted, dev, arm64 ]
runs-on: [ self-hosted, large-arm64 ]

env:
# Use release build only, to have less debug info around
Expand Down Expand Up @@ -232,20 +232,20 @@ jobs:

- name: Run cargo build
run: |
mold -run cargo build --locked $CARGO_FLAGS $CARGO_FEATURES --bins --tests
mold -run cargo build --locked $CARGO_FLAGS $CARGO_FEATURES --bins --tests -j$(nproc)

- name: Run cargo test
env:
NEXTEST_RETRIES: 3
run: |
cargo nextest run $CARGO_FEATURES
cargo nextest run $CARGO_FEATURES -j$(nproc)

# Run separate tests for real S3
export ENABLE_REAL_S3_REMOTE_STORAGE=nonempty
export REMOTE_STORAGE_S3_BUCKET=neon-github-ci-tests
export REMOTE_STORAGE_S3_REGION=eu-central-1
# Avoid `$CARGO_FEATURES` since there's no `testing` feature in the e2e tests now
cargo nextest run --package remote_storage --test test_real_s3
cargo nextest run --package remote_storage --test test_real_s3 -j$(nproc)

# Run separate tests for real Azure Blob Storage
# XXX: replace region with `eu-central-1`-like region
Expand All @@ -255,12 +255,12 @@ jobs:
export REMOTE_STORAGE_AZURE_CONTAINER="${{ vars.REMOTE_STORAGE_AZURE_CONTAINER }}"
export REMOTE_STORAGE_AZURE_REGION="${{ vars.REMOTE_STORAGE_AZURE_REGION }}"
# Avoid `$CARGO_FEATURES` since there's no `testing` feature in the e2e tests now
cargo nextest run --package remote_storage --test test_real_azure
cargo nextest run --package remote_storage --test test_real_azure -j$(nproc)

check-codestyle-rust-arm:
needs: [ check-permissions, build-build-tools-image ]
timeout-minutes: 90
runs-on: [ self-hosted, dev, arm64 ]
runs-on: [ self-hosted, large-arm64 ]

container:
image: ${{ needs.build-build-tools-image.outputs.image }}
Expand All @@ -269,6 +269,11 @@ jobs:
password: ${{ secrets.NEON_DOCKERHUB_PASSWORD }}
options: --init

strategy:
fail-fast: false
matrix:
build_type: [ debug, release ]

steps:
- name: Fix git ownership
run: |
Expand Down Expand Up @@ -305,31 +310,35 @@ jobs:
exit 1
fi
echo "CLIPPY_COMMON_ARGS=${CLIPPY_COMMON_ARGS}" >> $GITHUB_ENV

- name: Run cargo clippy (debug)
if: matrix.build_type == 'debug'
run: cargo hack --feature-powerset clippy $CLIPPY_COMMON_ARGS
- name: Run cargo clippy (release)
if: matrix.build_type == 'release'
run: cargo hack --feature-powerset clippy --release $CLIPPY_COMMON_ARGS

- name: Check documentation generation
run: cargo doc --workspace --no-deps --document-private-items
if: matrix.build_type == 'release'
run: cargo doc --workspace --no-deps --document-private-items -j$(nproc)
env:
RUSTDOCFLAGS: "-Dwarnings -Arustdoc::private_intra_doc_links"

# Use `${{ !cancelled() }}` to run quck tests after the longer clippy run
- name: Check formatting
if: ${{ !cancelled() }}
if: ${{ !cancelled() && matrix.build_type == 'release' }}
run: cargo fmt --all -- --check

# https://github.com/facebookincubator/cargo-guppy/tree/bec4e0eb29dcd1faac70b1b5360267fc02bf830e/tools/cargo-hakari#2-keep-the-workspace-hack-up-to-date-in-ci
- name: Check rust dependencies
if: ${{ !cancelled() }}
if: ${{ !cancelled() && matrix.build_type == 'release' }}
run: |
cargo hakari generate --diff # workspace-hack Cargo.toml is up-to-date
cargo hakari manage-deps --dry-run # all workspace crates depend on workspace-hack

# https://github.com/EmbarkStudios/cargo-deny
- name: Check rust licenses/bans/advisories/sources
if: ${{ !cancelled() }}
if: ${{ !cancelled() && matrix.build_type == 'release' }}
run: cargo deny check

gather-rust-build-stats:
Expand All @@ -338,7 +347,7 @@ jobs:
contains(github.event.pull_request.labels.*.name, 'run-extra-build-stats') ||
contains(github.event.pull_request.labels.*.name, 'run-extra-build-*') ||
github.ref_name == 'main'
runs-on: [ self-hosted, gen3, large ]
runs-on: [ self-hosted, large ]
container:
image: ${{ needs.build-build-tools-image.outputs.image }}
credentials:
Expand Down Expand Up @@ -369,7 +378,7 @@ jobs:
run: make walproposer-lib -j$(nproc)

- name: Produce the build stats
run: cargo build --all --release --timings
run: cargo build --all --release --timings -j$(nproc)

- name: Upload the build stats
id: upload-stats
Expand Down
Loading
Loading