kv/kvserver: TestDecommission failed

[cockroach-teamcity]

kv/kvserver.TestDecommission failed with artifacts on master @ 3aa37f1c499f7c73dd3fa178cc5e2687e9e23311:

=== RUN   TestDecommission
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestDecommission2033654041
    test_log_scope.go:80: use -show-logs to present logs inline

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=bazel,gss

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-15881

[tbg]

=== RUN   TestDecommission
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestDecommission2033654041
    test_log_scope.go:80: use -show-logs to present logs inline
panic: pebble: inconsistent reference count: 1

goroutine 350532 [running]:
github.com/cockroachdb/pebble.(*flushableEntry).readerRef(...)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/flushable.go:64
github.com/cockroachdb/pebble.(*DB).updateReadStateLocked(0xc009450000, 0x0, 0x0)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/read_state.go:92 +0x2a8
github.com/cockroachdb/pebble.(*DB).makeRoomForWrite(0xc009450000, 0x0)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:1883 +0xfd5
github.com/cockroachdb/pebble.(*DB).maybeScheduleDelayedFlush.func1()
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1362 +0x253
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleDelayedFlush
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1336 +0x10f
I220413 06:53:23.459424 1 (gostd) testmain.go:1568  [-] 1  Test //pkg/kv/kvserver:kvserver_test exited with error code 2

Adding GA-blocker cc @cockroachdb/storage

[jbowens]

This error almost always indicates something above Pebble is failing to close an Iterator.

[nicktrav]

Unfortunately, this didn't seem trivially reproducible - I ran for ~2 hours on my gceworker with the following, at the same SHA this test failed at, and didn't get any hits:

$ bazel test pkg/kv/kvserver:all \
  --test_env=GOTRACEBACK=all \
  --test_filter=TestDecommission$ \
  --test_sharding_strategy=disabled \
  --test_output errors \
  --run_under '@com_github_cockroachdb_stress//:stress -maxruns 1000 -maxtime 1h0m0s -maxfails 1 -p 4' \
  --define gotags=bazel,gss \
  --nocache_test_results \
  --test_output streamed

I think cockroachdb/pebble#1597 will help with debugging this type of issue eventually, though it's basically a thought experiment at this point.

[cockroach-teamcity]

kv/kvserver.TestDecommission failed with artifacts on master @ 0add8527e82d0ae4987ea7105778c7aa6e43bd3a:

=== RUN   TestDecommission
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/751d67000aac5f3394c2369309253f02/logTestDecommission4079857731
    test_log_scope.go:80: use -show-logs to present logs inline

Help

See also: How To Investigate a Go Test Failure (internal)
Parameters in this failure:

• TAGS=bazel,gss

This test on roachdash | Improve this report!

[erikgrinaker]

Above was same failure mode:

panic: pebble: inconsistent reference count: 1

goroutine 344716 [running]:
github.com/cockroachdb/pebble.(*flushableEntry).readerRef(...)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/flushable.go:64
github.com/cockroachdb/pebble.(*DB).updateReadStateLocked(0xc000748000, 0x0, 0x0)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/read_state.go:92 +0x2a8
github.com/cockroachdb/pebble.(*DB).makeRoomForWrite(0xc000748000, 0x0)
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:1883 +0xfd5
github.com/cockroachdb/pebble.(*DB).maybeScheduleDelayedFlush.func1()
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1362 +0x253
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleDelayedFlush
	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/compaction.go:1336 +0x10f
I220423 07:45:47.579376 1 (gostd) testmain.go:1394  [-] 1  Test //pkg/kv/kvserver:kvserver_test exited with error code 2

[github-actions]

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.