Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bench/rttanalysis: TestBenchmarkExpectation failed #74396

Closed
cockroach-teamcity opened this issue Jan 3, 2022 · 9 comments · Fixed by #75599
Closed

bench/rttanalysis: TestBenchmarkExpectation failed #74396

cockroach-teamcity opened this issue Jan 3, 2022 · 9 comments · Fixed by #75599
Assignees
@ajwerner
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ d19fe5e61ae19b53cda5351449fb9172c2ccec63:

--- FAIL: TestBenchmarkExpectation (23.90s)
=== RUN   TestBenchmarkExpectation/Truncate/truncate_2_column_1_rows
=== CONT  TestBenchmarkExpectation/Truncate/truncate_2_column_1_rows
    rtt_analysis_bench.go:152: Truncate/truncate_2_column_1_rows: got 25, expected 23-24
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation3462684652/Truncate_truncate_2_column_1_rows.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Truncate/truncate_2_column_1_rows (20.04s)
=== RUN   TestBenchmarkExpectation/Truncate
    --- FAIL: TestBenchmarkExpectation/Truncate (22.20s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jan 3, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Jan 3, 2022
@ajwerner ajwerner self-assigned this Jan 4, 2022
@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 12, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ 5aefc070d1e7f5bfd860eae24dea2e7726dc6d8c:

--- FAIL: TestBenchmarkExpectation (30.66s)
=== RUN   TestBenchmarkExpectation/DropSequence/drop_3_sequences
=== CONT  TestBenchmarkExpectation/DropSequence/drop_3_sequences
    rtt_analysis_bench.go:152: DropSequence/drop_3_sequences: got 31, expected 34
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation2760209602/DropSequence_drop_3_sequences.jaeger.json
        --- FAIL: TestBenchmarkExpectation/DropSequence/drop_3_sequences (29.46s)
=== RUN   TestBenchmarkExpectation/DropSequence
    --- FAIL: TestBenchmarkExpectation/DropSequence (29.47s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ bed5b793f9ed223b8133fad72f401ae8d02e8771:

    validate_benchmark_data.go:120: success: expected AlterTableAddCheckConstraint/alter_table_add_1_check_constraint to perform KV lookups in [18, 19], got 18
    validate_benchmark_data.go:120: success: expected AlterRole/alter_role_with_3_options to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:120: success: expected Truncate/truncate_1_column_0_rows to perform KV lookups in [23, 24], got 24
    validate_benchmark_data.go:120: success: expected AlterTableSplit/alter_table_split_at_2_values to perform KV lookups in [21, 22], got 21
    validate_benchmark_data.go:120: success: expected AlterTableSplit/alter_table_split_at_1_value to perform KV lookups in [15, 16], got 15
    validate_benchmark_data.go:120: success: expected DropDatabase/drop_database_3_tables to perform KV lookups in [42, 47], got 47
    validate_benchmark_data.go:120: success: expected DropSequence/drop_1_sequence to perform KV lookups in [22, 22], got 22
    validate_benchmark_data.go:120: success: expected Truncate/truncate_1_column_2_rows to perform KV lookups in [23, 24], got 24
    validate_benchmark_data.go:120: success: expected AlterTableUnsplit/alter_table_unsplit_at_2_values to perform KV lookups in [13, 13], got 13
    validate_benchmark_data.go:120: success: expected AlterTableAddCheckConstraint/alter_table_add_3_check_constraints to perform KV lookups in [18, 19], got 18
    validate_benchmark_data.go:120: success: expected Grant/grant_all_on_3_tables to perform KV lookups in [23, 24], got 23
    validate_benchmark_data.go:120: success: expected ORMQueries/pg_class to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected AlterTableAddForeignKey/alter_table_add_1_foreign_key to perform KV lookups in [23, 23], got 23
    validate_benchmark_data.go:120: success: expected AlterTableConfigureZone/alter_table_configure_zone_5_replicas to perform KV lookups in [24, 24], got 24
    validate_benchmark_data.go:120: success: expected CreateRole/create_role_with_2_options to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:120: success: expected Grant/grant_all_on_1_table to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:120: success: expected ORMQueries/django_table_introspection_1_table to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected AlterTableDropConstraint/alter_table_drop_3_check_constraints to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:120: success: expected DropView/drop_3_views to perform KV lookups in [39, 39], got 39
    validate_benchmark_data.go:120: success: expected DropRole/drop_2_roles to perform KV lookups in [34, 34], got 34
    validate_benchmark_data.go:120: success: expected AlterTableAddColumn/alter_table_add_3_columns to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:120: success: expected ORMQueries/pg_type to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected ORMQueries/django_table_introspection_4_tables to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected Revoke/revoke_all_on_3_tables to perform KV lookups in [23, 23], got 23
    validate_benchmark_data.go:120: success: expected ORMQueries/information_schema._pg_index_position to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:120: success: expected ORMQueries/has_table_privilege_real_table to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected ORMQueries/activerecord_type_introspection_query to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected DropTable/drop_1_table to perform KV lookups in [20, 24], got 24
    validate_benchmark_data.go:120: success: expected RevokeRole/revoke_1_role to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:120: success: expected GrantRole/grant_2_roles to perform KV lookups in [20, 20], got 20
    validate_benchmark_data.go:120: success: expected ORMQueries/pg_namespace to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:120: success: expected Revoke/revoke_all_on_1_table to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:120: success: expected ORMQueries/has_table_privilege_virtual_table to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected ORMQueries/has_column_privilege_using_attnum to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected GrantRole/grant_1_role to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:120: success: expected Truncate/truncate_1_column_1_row to perform KV lookups in [23, 24], got 24
    validate_benchmark_data.go:120: success: expected ORMQueries/django_table_introspection_8_tables to perform KV lookups in [2, 2], got 2
    validate_benchmark_data.go:120: success: expected Revoke/revoke_all_on_2_tables to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:120: success: expected RevokeRole/revoke_2_roles to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:74: see the --rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:107: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation372422407
--- FAIL: TestBenchmarkExpectation (35.89s)
=== RUN   TestBenchmarkExpectation/DropSequence/drop_2_sequences
=== CONT  TestBenchmarkExpectation/DropSequence/drop_2_sequences
    rtt_analysis_bench.go:152: DropSequence/drop_2_sequences: got 25, expected 28
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation372422407/DropSequence_drop_2_sequences.jaeger.json
        --- FAIL: TestBenchmarkExpectation/DropSequence/drop_2_sequences (28.34s)
=== RUN   TestBenchmarkExpectation/DropSequence
    --- FAIL: TestBenchmarkExpectation/DropSequence (31.15s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ 72655fd4730fadb0ddacd8a5513a5d2d9121cbc6:

--- FAIL: TestBenchmarkExpectation (39.65s)
=== RUN   TestBenchmarkExpectation/Truncate/truncate_1_column_2_rows
=== CONT  TestBenchmarkExpectation/Truncate/truncate_1_column_2_rows
    rtt_analysis_bench.go:152: Truncate/truncate_1_column_2_rows: got 25, expected 23-24
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation1998960504/Truncate_truncate_1_column_2_rows.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Truncate/truncate_1_column_2_rows (32.06s)
=== RUN   TestBenchmarkExpectation/Truncate
    --- FAIL: TestBenchmarkExpectation/Truncate (34.30s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ 6aa6c727b1d990bc0e3e8fbc36e25fc358ba39c1:

--- FAIL: TestBenchmarkExpectation (35.41s)
=== RUN   TestBenchmarkExpectation/DropDatabase/drop_database_0_tables
=== CONT  TestBenchmarkExpectation/DropDatabase/drop_database_0_tables
    rtt_analysis_bench.go:152: DropDatabase/drop_database_0_tables: got 24, expected 26
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation1634133125/DropDatabase_drop_database_0_tables.jaeger.json
        --- FAIL: TestBenchmarkExpectation/DropDatabase/drop_database_0_tables (27.32s)
=== RUN   TestBenchmarkExpectation/DropDatabase
    --- FAIL: TestBenchmarkExpectation/DropDatabase (32.76s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@irfansharif
Copy link
Contributor

Speculative analysis: recent failures coincide with #73876 landing. A possible culprit might be the same sort as #75304 (test not expecting stale distsender caches). Not assigning it to myself to investigate/fix (working through other fallout at the moment), but will get to it if this nobody gets to it before.

@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ f8a330b7f750fb2c5faf4109c80d99ba9aac3070:

--- FAIL: TestBenchmarkExpectation (45.84s)
=== RUN   TestBenchmarkExpectation/DropDatabase
    --- FAIL: TestBenchmarkExpectation/DropDatabase (41.20s)
=== RUN   TestBenchmarkExpectation/DropDatabase/drop_database_0_tables
=== CONT  TestBenchmarkExpectation/DropDatabase/drop_database_0_tables
    rtt_analysis_bench.go:152: DropDatabase/drop_database_0_tables: got 22, expected 24
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation727821020/DropDatabase_drop_database_0_tables.jaeger.json
        --- FAIL: TestBenchmarkExpectation/DropDatabase/drop_database_0_tables (41.05s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ 66d7d6884ff86eddb9a782a3670f7154ffa181c6:

--- FAIL: TestBenchmarkExpectation (39.49s)
=== RUN   TestBenchmarkExpectation/DropDatabase/drop_database_0_tables
=== CONT  TestBenchmarkExpectation/DropDatabase/drop_database_0_tables
    rtt_analysis_bench.go:152: DropDatabase/drop_database_0_tables: got 22, expected 24
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation3461407419/DropDatabase_drop_database_0_tables.jaeger.json
        --- FAIL: TestBenchmarkExpectation/DropDatabase/drop_database_0_tables (36.77s)
=== RUN   TestBenchmarkExpectation/DropDatabase
    --- FAIL: TestBenchmarkExpectation/DropDatabase (37.00s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@irfansharif
Copy link
Contributor

rtt_analysis_bench.go:152: DropDatabase/drop_database_0_tables: got 22, expected 24
rtt_analysis_bench.go:152: DropSequence/drop_2_sequences: got 25, expected 28
rtt_analysis_bench.go:152: DropSequence/drop_3_sequences: got 31, expected 34

Only had the most cursory look here. The failures do line up with when #73876 landed, but to the best of my knowledge, we didn't make anything faster. I'm not sure what to make of the variances above, what's contributing to it, and how #73876 fits in. There's more scanning of the system.descriptors now with the reconciliation job, so maybe distsender caches are pre-emptively primed? Or descriptors leased before the benchmarks expected them to be, reducing RTTs? (#75304 may be of interest to readers.)

I'll just let this issue sit on SQL Schema's plate, though perhaps noting that it is indeed flakier than before and perhaps should be skipped and/or have the expectations file be more permissive wrt RTT ranges.

ajwerner added a commit to ajwerner/cockroach that referenced this issue Jan 27, 2022
@ajwerner
bench/rttanalysis: skip the first iteration, take average
96c87bb 
We've had some flakes lately. One hypothesis I had was that we were seeing a
restart. Indeed when I added an extra iteration, it shook out the numbers we
were seeing. I didn't dig very deeply. I imagine there's something interesting
there. However, I just want stable tests.

Fixes cockroachdb#74396

Release note: None
@ajwerner ajwerner mentioned this issue Jan 27, 2022
Merged
@cockroach-teamcity
Copy link
Member Author

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on master @ 659fe193978a65187fcce81e985e14f9e2de9606:

--- FAIL: TestBenchmarkExpectation (52.04s)
=== RUN   TestBenchmarkExpectation/DropDatabase/drop_database_2_tables
=== CONT  TestBenchmarkExpectation/DropDatabase/drop_database_2_tables
    rtt_analysis_bench.go:152: DropDatabase/drop_database_2_tables: got 34, expected 36-42
    rtt_analysis_bench.go:158: wrote jaeger trace to /go/src/github.com/cockroachdb/cockroach/artifacts/logTestBenchmarkExpectation4154029015/DropDatabase_drop_database_2_tables.jaeger.json
        --- FAIL: TestBenchmarkExpectation/DropDatabase/drop_database_2_tables (50.47s)
=== RUN   TestBenchmarkExpectation/DropDatabase
    --- FAIL: TestBenchmarkExpectation/DropDatabase (50.49s)
Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

  • GOFLAGS=-json

Same failure on other branches

This test on roachdash | Improve this report!

@irfansharif irfansharif mentioned this issue Jan 27, 2022
Merged
@irfansharif irfansharif mentioned this issue Jan 28, 2022
Closed
23 tasks
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bench/rttanalysis: skip the first iteration, take average ajwerner/cockroach
3 participants
@ajwerner @irfansharif @cockroach-teamcity