Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: sync between benchies #8919

Merged
merged 6 commits into from
Sep 5, 2024
Merged

build: sync between benchies #8919

merged 6 commits into from
Sep 5, 2024

Conversation

koivunej
Copy link
Member

@koivunej koivunej commented Sep 4, 2024

Sometimes, the benchmarks fail to start up pageserver in 10s without any obvious reason. Benchmarks run sequentially on otherwise idle runners. Try running sync(2) after each bench to force a cleaner slate.

Implement this via:

  • SYNC_AFTER_EACH_TEST environment variable enabled autouse fixture
  • autouse fixture seems to be outermost fixture, so it works as expected
  • set SYNC_AFTER_EACH_TEST=true for benchmarks in build_and_test workflow

Evidence: https://neon-github-public-dev.s3.amazonaws.com/reports/main/10678984691/index.html#suites/5008d72a1ba3c0d618a030a938fc035c/1210266507534c0f/

@koivunej koivunej added the run-benchmarks Indicates to the CI that benchmarks should be run for PR marked with this label label Sep 4, 2024
@koivunej koivunej requested a review from bayandin September 4, 2024 16:25
test_runner/fixtures/compare_fixtures.py Outdated Show resolved Hide resolved
test_runner/fixtures/compare_fixtures.py Outdated Show resolved Hide resolved
test_runner/fixtures/compare_fixtures.py Outdated Show resolved Hide resolved
Copy link

github-actions bot commented Sep 4, 2024

3917 tests run: 3796 passed, 1 failed, 120 skipped (full report)


Failures on Postgres 16

  • test_basebackup_with_high_slru_count[github-actions-selfhosted-10-13-30]: release-x86-64
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_basebackup_with_high_slru_count[release-pg16-github-actions-selfhosted-10-13-30]"
Flaky tests (6)

Postgres 16

Postgres 15

Code coverage* (full report)

  • functions: 32.3% (7430 of 23006 functions)
  • lines: 50.5% (60177 of 119170 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
3072e37 at 2024-09-05T14:23:57.319Z :recycle:

@koivunej koivunej force-pushed the joonas/sync_between_benchies branch from 5ad05b9 to 7617274 Compare September 4, 2024 19:25
@koivunej
Copy link
Member Author

koivunej commented Sep 4, 2024

Since the first run didn't fail but also didn't sync(1) due to the wrong path, while at the same time a bench run on main again failed with the startup timeout.

I think it would be best to merge this fast as soon as we have evidence of sync being run for a non-zero amount of time and see if it changes anything with the benches on main.

Assuming the current run goes well, could @bayandin add a fix for the comment? Then, let's have a new run and merge.

No failures with the third run either.

@koivunej koivunej requested a review from bayandin September 5, 2024 08:14
@koivunej koivunej enabled auto-merge (squash) September 5, 2024 11:27
test_runner/fixtures/compare_fixtures.py Outdated Show resolved Hide resolved
test_runner/fixtures/compare_fixtures.py Outdated Show resolved Hide resolved
@koivunej koivunej disabled auto-merge September 5, 2024 12:16
@koivunej koivunej enabled auto-merge (squash) September 5, 2024 12:18
@koivunej koivunej merged commit efe03d5 into main Sep 5, 2024
70 of 71 checks passed
@koivunej koivunej deleted the joonas/sync_between_benchies branch September 5, 2024 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-benchmarks Indicates to the CI that benchmarks should be run for PR marked with this label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants