Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv/kvnemesis: TestKVNemesisMultiNode failed #94876

Closed
cockroach-teamcity opened this issue Jan 7, 2023 · 17 comments · Fixed by #99017
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #94876

cockroach-teamcity opened this issue Jan 7, 2023 · 17 comments · Fixed by #99017
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 7, 2023

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 7594fb2d1438dd5ed8f80a887b8d60b6cdd296ed:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2973024937
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:179: seed: 5290695158787770555
    kvnemesis_test.go:123: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis395234591
    kvnemesis.go:165: range 59 (/Table/100/"c7e56327021899f7") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1673079694631385519 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:3456 SysCount:11 AbortSpanBytes:342}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1673079694631385519 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:56 SysCount:1 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis395234591/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis395234591/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis395234591/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis395234591/kvs-scan.txt
    kvnemesis_test.go:206: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1075/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:206
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1075/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:159
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2973024937
--- FAIL: TestKVNemesisMultiNode (19.93s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-23198

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jan 7, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Jan 7, 2023
@blathers-crl blathers-crl bot added the T-kv KV Team label Jan 7, 2023
@erikgrinaker
Copy link
Contributor

Similar to #93890, we should expand the exception in #93897.

@erikgrinaker erikgrinaker self-assigned this Jan 9, 2023
@erikgrinaker erikgrinaker added T-kv-replication and removed T-kv KV Team labels Jan 9, 2023
@blathers-crl
Copy link

blathers-crl bot commented Jan 9, 2023

cc @cockroachdb/replication

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 51be9f048a59be0d4353498b447d162145384d13:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1144817863
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:179: seed: 5803121901063040129
    kvnemesis_test.go:123: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2770816698
    kvnemesis.go:165: range 58 (/Table/57) RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1676450146042019606 IntentAge:0 GCBytesAge:70602 LiveBytes:106 LiveCount:2 KeyBytes:2406 KeyCount:46 ValBytes:1975 ValCount:120 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:3 RangeKeyBytes:219 RangeValCount:9 RangeValBytes:93 SysBytes:8854 SysCount:27 AbortSpanBytes:1692}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1676450146042019606 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:8 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2770816698/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2770816698/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2770816698/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2770816698/kvs-scan.txt
    kvnemesis_test.go:206: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2469/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:206
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2469/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:159
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1144817863
--- FAIL: TestKVNemesisMultiNode (47.92s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ e4924e2b9be4a36d466beab53a80df9241df4783:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2364705987
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 6164690332929010067
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2998365209
    kvnemesis.go:165: range 99 (/Table/100/"fd644370488fc669") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1678691595996490343 IntentAge:0 GCBytesAge:11563 LiveBytes:106 LiveCount:2 KeyBytes:396 KeyCount:12 ValBytes:231 ValCount:12 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:14 RangeKeyBytes:840 RangeValCount:28 RangeValBytes:296 SysBytes:879 SysCount:6 AbortSpanBytes:85}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1678691595996490343 IntentAge:0 GCBytesAge:-720 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:-1 RangeKeyBytes:-60 RangeValCount:-2 RangeValBytes:-20 SysBytes:0 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2998365209/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2998365209/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2998365209/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2998365209/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2391/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2391/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2364705987
--- FAIL: TestKVNemesisMultiNode (19.50s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@erikgrinaker
Copy link
Contributor

Last failure is similar to #98473, and caused by #98426 and #94140.

craig bot pushed a commit that referenced this issue Mar 13, 2023
98472: changefeedccl: enable TestChangefeedPropagatesTerminalError r=samiskin a=samiskin

Fixes: #95057

The test TestChangefeedPropagatesTerminalError no longer seems to flake after running it under stress on a GCE worker for a while, so this change re-enables it.

Release note: None

98475: kvnemesis: disable `AddSSTable` range keys r=erikgrinaker a=erikgrinaker

These trigger MVCC stats bugs in `CheckSSTConflicts`.

Touches #94141.
Touches #98473.
Touches #94876.

Epic: none
Release note: None

Co-authored-by: Shiranka Miskin <shiranka.miskin@gmail.com>
Co-authored-by: Erik Grinaker <grinaker@cockroachlabs.com>
itsbilal added a commit to itsbilal/cockroach that referenced this issue Mar 13, 2023
A few additional fixes around CheckSSTConflicts, stats
calculations, and Next()ing logic, caught by kvnemesis.
Hopefully the last of its kind.

Fixes cockroachdb#94141.
Fixes cockroachdb#98473.
Informs cockroachdb#94876.

Epic: none

Release note: None
itsbilal added a commit to itsbilal/cockroach that referenced this issue Mar 13, 2023
A few additional fixes around CheckSSTConflicts, stats
calculations, and Next()ing logic, caught by kvnemesis.
Hopefully the last of its kind.

Also re-enable kvnemesis testing for range keys in
AddSSTable, reverting cockroachdb#98475.

Fixes cockroachdb#94141.
Fixes cockroachdb#98473.
Informs cockroachdb#94876.

Epic: none

Release note: None
craig bot pushed a commit that referenced this issue Mar 14, 2023
98368: multiregionccl,server: use cached sqlliveness.Reader, deflake ColdStartLatencyTest r=ajwerner a=ajwerner

#### multitenantccl: deflake ColdStartLatencyTest
This test was flakey due to the closed timestamp sometimes not leading far
for global tables due to overload, and due to a cached liveness reader
not being used in distsql. The former was fixed in previous commits. The
latter is fixed here.

Fixes: #96334


#### sql: use CachedReader for uses with sqlinstance and the sql builtins
The CachedReader won't block, which in multi-region clusters is good. It will
mean that in some cases, it'll state that a sessions is alive when it most
certainly is not. Currently, nobody needs synchronous semantics.

This is a major part of fixing the TestColdStartLatency as sometimes
distsql planning would block. That's not acceptable -- the idea that
query physical planning can need to wait for a cross-region RPC is
unacceptable.

#### sqlliveness: re-arrange APIs to clarify when the API was blocking

By "default" the implementation of Reader was blocking and there was a method
to get a handle to a non-blocking CachedReader(). This asymmetry does not aid
understandability. The non-blocking reader came later, but it is generally the
more desirable interface.

Release note: None

98519: storage: More CheckSSTConflicts fixes r=erikgrinaker a=itsbilal

A few additional fixes around CheckSSTConflicts, stats calculations, and Next()ing logic, caught by kvnemesis. Hopefully the last of its kind.

Also re-enable kvnemesis testing for range keys in AddSSTable, reverting #98475.

Fixes #94141.
Fixes #98473.
Informs #94876.

Epic: none

Release note: None

98567: backupccl: use correct version gate for restore checkpointing r=adityamaru a=msbutler

PR #97862 introduced a subtle bug which allowed the new restore checkpointing policy to take effect before the 23_1 migrations occured. This patch ensures the new policy only takes effect after all migrations occur.

Release note: None

Epic: None

Co-authored-by: ajwerner <awerner32@gmail.com>
Co-authored-by: Bilal Akhtar <bilal@cockroachlabs.com>
Co-authored-by: Michael Butler <butler@cockroachlabs.com>
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 53dbb86acb1d48309530181b94838faf937084d3:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3438109679
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 7643290107646314751
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3597272977
    kvnemesis.go:165: range 97 (/Table/100/"cddb5893952b3bd4") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679213275756613430 IntentAge:0 GCBytesAge:142049 LiveBytes:371 LiveCount:7 KeyBytes:3171 KeyCount:71 ValBytes:2195 ValCount:140 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:11 RangeKeyBytes:714 RangeValCount:28 RangeValBytes:308 SysBytes:3524 SysCount:21 AbortSpanBytes:1040}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679213275756613430 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3597272977/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3597272977/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3597272977/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3597272977/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3438109679
--- FAIL: TestKVNemesisMultiNode (73.67s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 53dbb86acb1d48309530181b94838faf937084d3:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode908028899
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 3951388066019291456
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2173246507
    kvnemesis.go:165: range 84 (/Table/100/"a9f01d4f1e9767ad") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679214154871948777 IntentAge:0 GCBytesAge:925 LiveBytes:0 LiveCount:0 KeyBytes:33 KeyCount:1 ValBytes:11 ValCount:1 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:51 RangeValCount:1 RangeValBytes:10 SysBytes:1255 SysCount:16 AbortSpanBytes:935}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679214154871948777 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2173246507/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2173246507/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2173246507/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2173246507/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode908028899
--- FAIL: TestKVNemesisMultiNode (22.60s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ de5b08a418390a803357ba4b74a55a4eec26e16b:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode4249152215
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 4174854140454642154
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4269141150
    kvnemesis.go:165: range 113 (/Table/100/"a1d598c9835d180a") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679299778512039327 IntentAge:0 GCBytesAge:3078 LiveBytes:0 LiveCount:0 KeyBytes:246 KeyCount:6 ValBytes:158 ValCount:10 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:60 RangeValCount:2 RangeValBytes:20 SysBytes:1706 SysCount:21 AbortSpanBytes:1382}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679299778512039327 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4269141150/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4269141150/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4269141150/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4269141150/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode4249152215
--- FAIL: TestKVNemesisMultiNode (24.04s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ b89fa2cc4bc1fb9447ab1009b34b6f354f9618f0:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3097352090
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 4512943326389223277
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2928488251
    kvnemesis.go:165: range 123 (/Table/100/"acfff0df4d282f49") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679383666350302551 IntentAge:0 GCBytesAge:11304 LiveBytes:0 LiveCount:0 KeyBytes:171 KeyCount:3 ValBytes:109 ValCount:9 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:69 RangeValCount:3 RangeValBytes:30 SysBytes:4769 SysCount:23 AbortSpanBytes:1487}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679383666350302551 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2928488251/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2928488251/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2928488251/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2928488251/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3097352090
--- FAIL: TestKVNemesisMultiNode (66.90s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ b89fa2cc4bc1fb9447ab1009b34b6f354f9618f0:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode606390385
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 4345960837954812798
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3684346812
    kvnemesis.go:165: range 94 (/Table/100/"ae6ee030e4149878") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679384863958683208 IntentAge:0 GCBytesAge:9321 LiveBytes:0 LiveCount:0 KeyBytes:651 KeyCount:11 ValBytes:536 ValCount:35 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:51 RangeValCount:1 RangeValBytes:10 SysBytes:2406 SysCount:25 AbortSpanBytes:1711}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679384863958683208 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3684346812/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3684346812/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3684346812/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3684346812/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode606390385
--- FAIL: TestKVNemesisMultiNode (24.04s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 78bb7696847e1d8abf5d6a199c6dd3414a568691:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2970276085
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 4666564691465668039
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1794424687
    kvnemesis.go:165: range 108 (/Table/100/"1e9a59f941568250") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679476127718336064 IntentAge:0 GCBytesAge:4167 LiveBytes:53 LiveCount:1 KeyBytes:216 KeyCount:4 ValBytes:183 ValCount:11 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:2 RangeKeyBytes:129 RangeValCount:5 RangeValBytes:55 SysBytes:1063 SysCount:12 AbortSpanBytes:595}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679476127718336064 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1794424687/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1794424687/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1794424687/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1794424687/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2970276085
--- FAIL: TestKVNemesisMultiNode (25.06s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 5493fdfec4e1762c4502fb2f5d42fd28292c9c9d:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode250515544
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 3704284822724010972
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2731013122
    kvnemesis.go:165: range 94 (/Table/100/"df7f758e1437ff8b") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679554542803220751 IntentAge:0 GCBytesAge:499 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:60 RangeValCount:2 RangeValBytes:20 SysBytes:1178 SysCount:9 AbortSpanBytes:344}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679554542803220751 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:165: range 96 (/Table/100/"cc14f12855cb7b5e") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679554543279349257 IntentAge:0 GCBytesAge:9713 LiveBytes:53 LiveCount:1 KeyBytes:1188 KeyCount:24 ValBytes:926 ValCount:57 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:6 RangeKeyBytes:369 RangeValCount:13 RangeValBytes:132 SysBytes:1548 SysCount:14 AbortSpanBytes:688}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679554543279349257 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2731013122/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2731013122/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2731013122/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2731013122/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 2
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode250515544
--- FAIL: TestKVNemesisMultiNode (15.22s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 19a6b804d3aff74d74619a75cac3b52338c7aa02:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2764319091
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 6076709081761455074
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis892199864
    kvnemesis.go:165: range 78 (/Table/100/"4794dd4eb027e7ec") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679643132766711808 IntentAge:0 GCBytesAge:7713 LiveBytes:0 LiveCount:0 KeyBytes:984 KeyCount:24 ValBytes:680 ValCount:40 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:11 RangeKeyBytes:678 RangeValCount:24 RangeValBytes:261 SysBytes:9523 SysCount:27 AbortSpanBytes:1634}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679643132766711808 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis892199864/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis892199864/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis892199864/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis892199864/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2764319091
--- FAIL: TestKVNemesisMultiNode (15.72s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 19a6b804d3aff74d74619a75cac3b52338c7aa02:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode441672708
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 300093096846407620
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1630625541
    kvnemesis.go:165: range 114 (/Table/100/"2e4486da9d58fd1e") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679644104954169621 IntentAge:0 GCBytesAge:8866 LiveBytes:0 LiveCount:0 KeyBytes:417 KeyCount:9 ValBytes:299 ValCount:19 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:51 RangeValCount:1 RangeValBytes:10 SysBytes:1641 SysCount:21 AbortSpanBytes:1329}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679644104954169621 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1630625541/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1630625541/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1630625541/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1630625541/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode441672708
--- FAIL: TestKVNemesisMultiNode (46.27s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 0fcc33bc2870961b9387999d8a9fed97fccbb2ae:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2626729611
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 4687563778299142964
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis734916401
    kvnemesis.go:165: error applying x.AdminSplit(ctx, tk(6275099482598792155)) // replica cannot hold lease: replica cannot hold lease
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis734916401/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis734916401/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis734916401/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis734916401/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode2626729611
--- FAIL: TestKVNemesisMultiNode (15.72s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ d107217dac5d817cc115bc0e97b7e53c0f2878bf:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3717913437
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 6316605110465706981
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2055983410
    kvnemesis.go:165: range 104 (/Table/100/"06dedfdd9ea97260") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679817209648949855 IntentAge:0 GCBytesAge:9082 LiveBytes:106 LiveCount:2 KeyBytes:549 KeyCount:9 ValBytes:460 ValCount:30 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:51 RangeValCount:1 RangeValBytes:11 SysBytes:3326 SysCount:25 AbortSpanBytes:1454}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679817209648949855 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2055983410/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2055983410/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2055983410/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2055983410/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3717913437
--- FAIL: TestKVNemesisMultiNode (20.50s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 2bd2c806ab3044569b09e0a205b5bc0452ad4e2b:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1084990637
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 9005252345550530387
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2111337116
    kvnemesis.go:165: range 65 (/Table/100/"684d770a9d8679e1") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679903063652324857 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:4775 SysCount:14 AbortSpanBytes:736}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679903063652324857 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-12 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2111337116/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2111337116/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2111337116/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2111337116/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:160
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1084990637
--- FAIL: TestKVNemesisMultiNode (45.68s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@craig craig bot closed this as completed in 3c3d2a5 Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants