Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bench/ddl_analysis: TestBenchmarkExpectation failed #61856

Closed
cockroach-teamcity opened this issue Mar 11, 2021 · 1 comment · Fixed by #61877
Closed

bench/ddl_analysis: TestBenchmarkExpectation failed #61856

cockroach-teamcity opened this issue Mar 11, 2021 · 1 comment · Fixed by #61877
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

(bench/ddl_analysis).TestBenchmarkExpectation failed on master@a86cf540219a82234187db11434ab664fe335f17:

BenchmarkRevokeRole/revoke_2_roles
BenchmarkRevokeRole/revoke_2_roles-16                                               	       1	 175696610 ns/op	        31.0 roundtrips
    ddl_analysis_bench.go:107: -- test log scope end --
BenchmarkORMQueries
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logBenchmarkORMQueries732589748
    test_log_scope.go:74: use -show-logs to present logs inline
BenchmarkORMQueries/django_table_introspection_1_table
BenchmarkORMQueries/django_table_introspection_1_table-16                           	       1	  37953698 ns/op	        10.0 roundtrips
BenchmarkORMQueries/django_table_introspection_4_tables
BenchmarkORMQueries/django_table_introspection_4_tables-16                          	       1	 114071576 ns/op	        13.0 roundtrips
BenchmarkORMQueries/django_table_introspection_8_tables
BenchmarkORMQueries/django_table_introspection_8_tables-16                          	       1	 125694843 ns/op	        17.0 roundtrips
BenchmarkORMQueries/activerecord_type_introspection_query
BenchmarkORMQueries/activerecord_type_introspection_query-16                        	       1	  33336729 ns/op	         5.00 roundtrips
    ddl_analysis_bench.go:107: -- test log scope end --
BenchmarkSystemDatabaseQueries
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logBenchmarkSystemDatabaseQueries631603339
    test_log_scope.go:74: use -show-logs to present logs inline
BenchmarkSystemDatabaseQueries/select_system.users_with_schema_name
BenchmarkSystemDatabaseQueries/select_system.users_with_schema_name-16              	       1	   3526672 ns/op	         3.00 roundtrips
BenchmarkSystemDatabaseQueries/select_system.users_without_schema_name
BenchmarkSystemDatabaseQueries/select_system.users_without_schema_name-16           	       1	  13879778 ns/op	         4.00 roundtrips
BenchmarkSystemDatabaseQueries/select_system.users_with_empty_database_name
BenchmarkSystemDatabaseQueries/select_system.users_with_empty_database_name-16      	       1	   4060221 ns/op	         3.00 roundtrips
    ddl_analysis_bench.go:107: -- test log scope end --
BenchmarkTruncate
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logBenchmarkTruncate701464068
    test_log_scope.go:74: use -show-logs to present logs inline
BenchmarkTruncate/truncate_1_column_0_rows
BenchmarkTruncate/truncate_1_column_0_rows-16                                       	       1	 169310072 ns/op	        22.0 roundtrips
BenchmarkTruncate/truncate_1_column_1_row
BenchmarkTruncate/truncate_1_column_1_row-16                                        	       1	 173882345 ns/op	        22.0 roundtrips
BenchmarkTruncate/truncate_1_column_2_rows
BenchmarkTruncate/truncate_1_column_2_rows-16                                       	       1	 199784519 ns/op	        22.0 roundtrips
BenchmarkTruncate/truncate_2_column_0_rows
BenchmarkTruncate/truncate_2_column_0_rows-16                                       	       1	 149220333 ns/op	        22.0 roundtrips
BenchmarkTruncate/truncate_2_column_1_rows
BenchmarkTruncate/truncate_2_column_1_rows-16                                       	       1	 124147490 ns/op	        22.0 roundtrips
BenchmarkTruncate/truncate_2_column_2_rows
BenchmarkTruncate/truncate_2_column_2_rows-16                                       	       1	  87393345 ns/op	        22.0 roundtrips
    ddl_analysis_bench.go:107: -- test log scope end --
BenchmarkVirtualTableQueries
    test_log_scope.go:73: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logBenchmarkVirtualTableQueries457240831
    test_log_scope.go:74: use -show-logs to present logs inline
BenchmarkVirtualTableQueries/select_crdb_internal.tables_with_1_fk
BenchmarkVirtualTableQueries/select_crdb_internal.tables_with_1_fk-16               	       1	  14032247 ns/op	         1.00 roundtrips
BenchmarkVirtualTableQueries/select_crdb_internal.invalid_objects_with_1_fk
BenchmarkVirtualTableQueries/select_crdb_internal.invalid_objects_with_1_fk-16      	       1	  15087125 ns/op	         1.00 roundtrips
    ddl_analysis_bench.go:107: -- test log scope end --
--- FAIL: TestBenchmarkExpectation (128.08s)

More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestBenchmarkExpectation PKG=./pkg/bench/ddl_analysis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 11, 2021
@ajwerner
Copy link
Contributor

Well that was fast!

craig bot pushed a commit that referenced this issue Mar 11, 2021
61807: sqlsmith: fix error handling in makeAlter and panic handling r=fqazi a=fqazi

Fixes: #61775

Previously, the error from ReloadSchemas was ignored
and the statement generator would blindly continue.
This lead to sqlsmith failing completely and panicking
inside roach test. To address this, this patch fixes
the error handling in this case and makes the sqlsmith
error handling more robust to panics.

Release note: None

61857: bench/ddl_analysis: skip because it's flakey r=ajwerner a=ajwerner

I stressed this thing a lot. I guess not enough.

Touch #61856. 

Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
Co-authored-by: Andrew Werner <awerner32@gmail.com>
ajwerner added a commit to ajwerner/cockroach that referenced this issue Mar 11, 2021
The root of the flakes is that while the coin was flipped such that
the parent process was not using metamorhpic testing, the child was.

Fixes cockroachdb#61856

Release note: None
craig bot pushed a commit that referenced this issue Mar 15, 2021
61406: sql: validate against array type usages when dropping enum values r=ajwerner a=arulajmani

Previously, when dropping an enum value, we weren't validating if the
enum value was used in a column of array type. This patch fixes the bug.

Fixes #60004

Release justification: bug fix to new functionality
Release note: None

61650: roachtest: exclude `lost+found` directory r=knz a=rickystewart

This directory is created by the filesystem and unowned file chunks are
put there by `fsck`. The directory and its contents aren't readable by
anyone except `root`, so this can cause the `du -c /mnt/data1` that
`roachtest` performs to fail -- add an `--exclude` to handle this.

We already ignore this directory in other contexts (for example, see
`pkg/storage/mvcc.go`).

Fixes #53663.

Release justification: Non-production code change
Release note: None

61788: importccl: unskip userfile benchmark r=pbardea a=adityamaru

I've run this ~20 times and it averages ~13s to run. I suspect the fixes to linked issues mentioned in #59126 might have mitigated this. Going to unskip due to lack of reproducibility.

Fixes: #59126

Release note: None

61828: contention: store contention events on non-SQL keys r=yuzefovich a=yuzefovich

Previously, whenever we tried to add a contention event on a non-SQL
key, it would encounter an error during decoding tableID/indexID pair,
and the event was dropped. This commit extends the contention registry
to additionally store information about contention on non-SQL keys. That
information is stored in two levels:
- on the top level, all `SingleNonSQLKeyContention` objects are ordered
  by their keys
- on the bottom level, all `SingleTxnContention` objects are ordered by the
  number of times that transaction was observed to contend with other
  transactions.

`SingleTxnContention` protobuf message is moved out of
`SingleKeyContention` and is reused for non-SQL keys. This commit also
updates the status server API response. I assume that no changes are
needed with regards to backwards compatibility since the original
version was merged just a few weeks ago, and we haven't had a beta
released since then.

Fixes: #60669.

Release note (sql change): CockroachDB now also stores the information
about contention on non-SQL keys.

61862: cliccl: add `load show backups` to display backup collection r=pbardea a=Elliebababa

Previously, users can list backups created by `BACKUP INTO`
with `SHOW BACKUP IN`in a sql session. But this listing task
can be also done offline without a running cluster.

This PR updates `load show` with `backups` subcommand, 
which allows users to list backups in a backup collection 
created by `BACKUP INTO`. 
With the same purpose as other `load show` subcommands, 
this update allows users to list backups without running 
`SHOW BACKUP IN` in a sql session.

see #61131 #61829 to checkout other `load show` subcommand.

Release note (cli change): Add `load show backups` to
display backup collection. Previously, users can list backups
created by `BACKUP INTO` via `SHOW BACKUP IN`in a sql
session. But this listing task can be also done offline without a
running cluster. Now, users are able to list backups in a collection
with `cockroach load show backups <collection_url>.

61877: bench/ddl_analysis: fix test for real r=ajwerner a=ajwerner

See individual commits. Last is the critical one. 

Fixes #61856.

61937: colexec: make vectorized stats concurrency safe r=yuzefovich a=yuzefovich

**colflow: clean up vectorized stats for rowexec processors**

Previously, the wrapped row-execution KV reading processors were
implementing `execinfra.KVReader` interface, but they were never used as
such, only the ColBatchScans would get used to retrieve the KV stats.
This is the case because the row-execution processors report their
execution stats themselves, and we don't want to duplicate that info.
This commit moves `KVReader` interface into `colexecop` package and now
only the ColBatchScans implement it. This allowed for some cleanup
around the vectorized stats code, but the main reason for performing
this change is that the contract of the interface will be modified by
the follow-up commit to mention the safety under concurrent usage, and
I didn't want to change the row-execution processors for that since the
relevant methods never get called anyway.

Additionally, this commit begins emitting of rows-read and bytes-read by
the zigzagJoiners and invertedJoiners to complete the metrics picture.

Release note: None

**colexec: make vectorized stats concurrency safe**

Previously, the collection of vectorized stats was not synchronized with
the operators themselves. Namely, it was possible to call methods like
`GetBytesRead` on the ColBatchScans and Inboxes from a different
goroutine (the root materializer or the outbox) than from the main
goroutine of the operator. This is now fixed by putting mutexes in place
and updating `colexecop.KVReader` interface to require concurrency-safe
implementations.

Fixes: #61899.

Release note: None

Co-authored-by: arulajmani <arulajmani@gmail.com>
Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
Co-authored-by: Aditya Maru <adityamaru@gmail.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Co-authored-by: elliebababa <ellie24.huang@gmail.com>
Co-authored-by: Andrew Werner <awerner32@gmail.com>
@craig craig bot closed this as completed in d9ab451 Mar 15, 2021
ajwerner added a commit to ajwerner/cockroach that referenced this issue Mar 22, 2021
The root of the flakes is that while the coin was flipped such that
the parent process was not using metamorhpic testing, the child was.

Fixes cockroachdb#61856

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants