Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestSpanBasedDependencyAnalyzer failed under stress #21812

Closed
cockroach-teamcity opened this issue Jan 26, 2018 · 4 comments
Closed

sql: TestSpanBasedDependencyAnalyzer failed under stress #21812

cockroach-teamcity opened this issue Jan 26, 2018 · 4 comments
Assignees
@nvanbenschoten
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.0

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/68ab61a20e5408febf3471dbef36f07c9ab87fb4

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=493778&tab=buildLog


I180126 08:49:49.953749 21144 server/status/runtime.go:219  [n1] runtime stats: 830 MiB RSS, 219 goroutines, 27 MiB/57 MiB/103 MiB GO alloc/idle/total, 38 MiB/88 MiB CGO alloc/total, 0.40cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:49:50.140025 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180126 08:49:52.681342 21142 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180126 08:50:19.652137 21294 storage/replica.go:1489  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s attempting to acquire lease
W180126 08:50:21.837798 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 24.5s
W180126 08:50:22.041976 21295 storage/replica.go:1489  [n1,s1,r3/1:/System/NodeLiveness{-Max}] have been waiting 1m0s attempting to acquire lease
W180126 08:50:34.631941 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180126 08:50:37.716054 21293 storage/replica.go:1489  [n1,s1,r1/1:/{Min-System/}] have been waiting 1m0s attempting to acquire lease
W180126 08:50:39.157285 21505 storage/replica.go:2757  [n1,s1,r19/1:/Table/2{2-3}] have been waiting 1m0s for proposing command GC [/Table/22,/Table/22/NULL)
W180126 08:50:44.639252 21608 storage/replica.go:1489  [n1,s1,r12/1:/Table/1{5-6}] have been waiting 1m0s attempting to acquire lease
I180126 08:50:44.888253 21144 server/status/runtime.go:219  [n1] runtime stats: 830 MiB RSS, 217 goroutines, 28 MiB/56 MiB/103 MiB GO alloc/idle/total, 38 MiB/88 MiB CGO alloc/total, 0.74cgo/sec, 0.21/0.02 %(u/s)time, 0.00 %gc (0x)
I180126 08:50:56.289306 21274 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956549.251535925,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1516956479.798439660,0 pro=1516956470.798523428,0
W180126 08:51:01.094766 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 20.4s
W180126 08:51:02.379399 19693 storage/replica.go:2757  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s for proposing command RequestLease [/Table/SystemConfigSpan/Start,/Min)
W180126 08:51:02.379398 19694 storage/replica.go:2757  [n1,s1,r12/1:/Table/1{5-6}] have been waiting 1m0s for proposing command RequestLease [/Table/15,/Min)
W180126 08:51:03.817565 21273 storage/store.go:3776  [n1,s1] handle raft ready: 96.8s [processed=1]
I180126 08:51:03.817624 21144 server/status/runtime.go:219  [n1] runtime stats: 831 MiB RSS, 220 goroutines, 28 MiB/57 MiB/103 MiB GO alloc/idle/total, 38 MiB/88 MiB CGO alloc/total, 0.56cgo/sec, 1.39/0.11 %(u/s)time, 0.15 %gc (1x)
W180126 08:51:03.817828 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180126 08:51:03.817913 21274 storage/store.go:3776  [n1,s1] handle raft ready: 59.6s [processed=1]
W180126 08:51:09.124610 21273 storage/store.go:3776  [n1,s1] handle raft ready: 4.7s [processed=0]
I180126 08:51:09.517783 21272 storage/replica_proposal.go:196  [n1,s1,r12/1:/Table/1{5-6}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956565.047971204,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1516956479.798439660,0 pro=1516956470.798523428,0
W180126 08:51:09.323775 19684 storage/replica.go:1489  [n1,s1,r4/1:/System/{NodeLive…-tsd}] have been waiting 1m0s attempting to acquire lease
W180126 08:51:09.623249 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180126 08:51:09.692999 21548 storage/replica.go:2757  [n1,s1,r1/1:/{Min-System/}] have been waiting 1m0s for proposing command RequestLease [/Min,/Min)
W180126 08:51:09.695370 19719 storage/replica.go:2757  [n1,s1,r3/1:/System/NodeLiveness{-Max}] have been waiting 1m0s for proposing command RequestLease [/System/NodeLiveness,/Min)
I180126 08:51:13.906895 21142 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180126 08:51:15.041281 21281 storage/store.go:3776  [n1,s1] handle raft ready: 72.5s [processed=1]
W180126 08:51:15.041296 21276 storage/store.go:3776  [n1,s1] handle raft ready: 70.0s [processed=1]
W180126 08:51:17.199408 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 4.3s
I180126 08:51:18.432826 21144 server/status/runtime.go:219  [n1] runtime stats: 831 MiB RSS, 217 goroutines, 17 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 4.21cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:51:18.657434 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180126 08:51:22.619339 21272 storage/store.go:3776  [n1,s1] handle raft ready: 83.5s [processed=1]
I180126 08:51:22.796692 21144 server/status/runtime.go:219  [n1] runtime stats: 831 MiB RSS, 217 goroutines, 17 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 6.06cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:51:23.942765 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 4.9s
W180126 08:51:27.082751 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180126 08:51:30.435565 21283 storage/replica_proposal.go:196  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956595.539505505,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1516956479.798439660,0 pro=1516956470.798523428,0
W180126 08:51:33.441669 21268 storage/store.go:3776  [n1,s1] handle raft ready: 4.0s [processed=0]
W180126 08:51:33.441668 21280 storage/store.go:3776  [n1,s1] handle raft ready: 4.0s [processed=0]
W180126 08:51:33.441671 21270 storage/store.go:3776  [n1,s1] handle raft ready: 4.0s [processed=0]
W180126 08:51:33.441666 21271 storage/store.go:3776  [n1,s1] handle raft ready: 4.0s [processed=0]
W180126 08:51:33.441678 21269 storage/store.go:3776  [n1,s1] handle raft ready: 4.0s [processed=0]
W180126 08:51:37.056330 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 5.4s
I180126 08:51:42.269258 21144 server/status/runtime.go:219  [n1] runtime stats: 831 MiB RSS, 217 goroutines, 17 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 2.80cgo/sec, 0.02/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:51:42.607636 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180126 08:51:43.606527 21284 storage/replica_proposal.go:196  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956662.144854110,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1516956479.798439660,0 pro=1516956470.798523428,0
W180126 08:51:51.224985 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 4.9s
I180126 08:51:55.954469 21142 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180126 08:51:56.194190 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180126 08:51:57.884113 21144 server/status/runtime.go:219  [n1] runtime stats: 832 MiB RSS, 216 goroutines, 18 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 1.32cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:52:03.360538 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 5.3s
W180126 08:52:05.426393 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180126 08:52:05.426908 21145 storage/replica.go:1489  [n1,s1,r5/1:/System/ts{d-e}] have been waiting 1m0s attempting to acquire lease
W180126 08:52:11.643809 21147 storage/replica.go:1489  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s attempting to acquire lease
W180126 08:52:11.643806 21294 storage/replica.go:1489  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s attempting to acquire lease
W180126 08:52:11.643804 21153 storage/replica.go:1489  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s attempting to acquire lease
I180126 08:52:11.647659 21144 server/status/runtime.go:219  [n1] runtime stats: 832 MiB RSS, 217 goroutines, 18 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 1.23cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:52:11.943704 21397 storage/replica.go:2757  [n1,s1,r4/1:/System/{NodeLive…-tsd}] have been waiting 1m0s for proposing command RequestLease [/System/NodeLivenessMax,/Min)
W180126 08:52:12.087615 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 5.2s
W180126 08:52:12.087859 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180126 08:52:12.419342 21144 server/status/runtime.go:219  [n1] runtime stats: 832 MiB RSS, 216 goroutines, 18 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 3.38cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:52:13.871635 21284 storage/store.go:3776  [n1,s1] handle raft ready: 51.3s [processed=1]
W180126 08:52:13.871853 21283 storage/store.go:3776  [n1,s1] handle raft ready: 64.1s [processed=1]
W180126 08:52:13.872257 21269 storage/store.go:3776  [n1,s1] handle raft ready: 30.9s [processed=1]
W180126 08:52:13.873873 21268 storage/store.go:3776  [n1,s1] handle raft ready: 31.0s [processed=1]
W180126 08:52:17.123604 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 4.5s
W180126 08:52:17.347255 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180126 08:52:26.154640 21204 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.154743 21197 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.154782 21199 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.154860 21203 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.154794 21202 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.155004 21196 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.154925 21195 storage/store.go:3776  [n1,s1] handle raft ready: 1.5s [processed=0]
W180126 08:52:26.383406 21608 storage/replica.go:1489  [n1,s1,r12/1:/Table/1{5-6}] have been waiting 1m0s attempting to acquire lease
W180126 08:52:26.383387 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 4.7s
W180126 08:52:30.449077 21151 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180126 08:52:30.829605 21144 server/status/runtime.go:219  [n1] runtime stats: 832 MiB RSS, 216 goroutines, 19 MiB/64 MiB/103 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 6.42cgo/sec, 0.02/0.01 %(u/s)time, 0.00 %gc (0x)
I180126 08:52:31.083537 21144 server/status/runtime.go:219  [n1] runtime stats: 804 MiB RSS, 215 goroutines, 19 MiB/35 MiB/75 MiB GO alloc/idle/total, 38 MiB/89 MiB CGO alloc/total, 0.87cgo/sec, 0.02/0.01 %(u/s)time, 0.00 %gc (0x)
W180126 08:52:31.634415 20059 storage/node_liveness.go:426  [n1,s1,r7/1:/Table/{SystemCon…-11}] slow heartbeat took 82.2s
W180126 08:52:31.635368 19697 storage/node_liveness.go:426  [n1,s1,r12/1:/Table/1{5-6}] slow heartbeat took 68.9s
W180126 08:52:31.635509 21702 storage/node_liveness.go:426  [n1,s1,r5/1:/System/ts{d-e}] slow heartbeat took 17.8s
E180126 08:52:31.687400 19697 storage/replica_range_lease.go:256  [n1,s1,r12/1:/Table/1{5-6}] heartbeat failed on epoch increment
E180126 08:52:31.688120 21702 storage/replica_range_lease.go:256  [n1,s1,r5/1:/System/ts{d-e}] heartbeat failed on epoch increment
W180126 08:52:31.691256 21746 storage/node_liveness.go:426  [n1,s1,r4/1:/System/{NodeLive…-tsd}] slow heartbeat took 17.8s
E180126 08:52:31.691519 21746 storage/replica_range_lease.go:256  [n1,s1,r4/1:/System/{NodeLive…-tsd}] heartbeat failed on epoch increment
W180126 08:52:31.692333 21151 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 1.2s
I180126 08:52:31.692581 21151 storage/node_liveness.go:362  [n1,hb] heartbeat failed on epoch increment; retrying
I180126 08:52:31.693415 21271 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956664.248395991,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956549.251535925,0
I180126 08:52:31.795207 21278 storage/replica_proposal.go:196  [n1,s1,r12/1:/Table/1{5-6}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956751.692171655,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956565.047971204,0
I180126 08:52:31.797064 20186 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956751.695380732,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956664.248395991,0
I180126 08:52:31.801284 21279 storage/replica_proposal.go:196  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956751.691717718,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956662.144854110,0
I180126 08:52:31.797377 21287 storage/replica_proposal.go:196  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956751.693844599,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1516956595.539505505,0
F180126 08:52:31.832703 19684 sqlmigrations/migrations.go:350  [n1] not enough time left on migration lease, terminating for safety
goroutine 19684 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x48d4301, 0x0, 0x0, 0x1b)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x3f08680, 0xc400000004, 0x39b81a1, 0x1b, 0x15e, 0xc4229fa230, 0x44)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x6c8
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3ce2e80, 0xc422a4fc20, 0x4, 0x2, 0x0, 0x0, 0xc4215efeb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x324
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3ce2e80, 0xc422a4fc20, 0x1, 0xc400000004, 0x0, 0x0, 0xc4215efeb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x3ce2e80, 0xc422a4fc20, 0xc4215efeb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x88
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations.func2(0x3ce2e80, 0xc422a4fc20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:350 +0x37c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc42271a120, 0x3ce2e80, 0xc422a4fc20, 0xc4200151d0, 0x29, 0x0, 0x0, 0xc4203630a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x14d

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc4202d4690, 0x2a21953, 0x1f, 0x2adf060, 0xc420663c01)
	/usr/local/go/src/testing/testing.go:790 +0x59b
testing.runTests.func1(0xc4202d4690)
	/usr/local/go/src/testing/testing.go:1004 +0xa8
testing.tRunner(0xc4202d4690, 0xc420663d50)
	/usr/local/go/src/testing/testing.go:746 +0x16d
testing.runTests(0xc42035ba00, 0x3cbb220, 0xd4, 0xd4, 0xc4200145a0)
	/usr/local/go/src/testing/testing.go:1002 +0x522
testing.(*M).Run(0xc420c09f20, 0x18)
	/usr/local/go/src/testing/testing.go:921 +0x207
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc420663f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:36 +0x15e
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/_test/_testmain.go:470 +0x1ce

goroutine 33 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3f08680)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1053 +0x8f
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0x13a

goroutine 34 [syscall, 7 minutes]:
os/signal.signal_recv(0x84ae01)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 37 [select, 4 minutes, locked to thread]:
runtime.gopark(0x2ae2ca0, 0x0, 0x29f565b, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0xfd
runtime.selectgo(0xc42004bf50, 0xc4200767e0)
	/usr/local/go/src/runtime/select.go:395 +0x12c3
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x1a7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 21189 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x317)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc420c6b800)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc420c6b800)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037cc40, 0xc42271a120, 0xc42037cc20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 20957 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3ce2e80, 0xc420db8060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:560 +0x1e6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4211ee3a0, 0xc42271a120, 0xc4229723c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21609 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0xc4229f4ebc)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4229f4eb0)
	/usr/local/go/src/sync/waitgroup.go:131 +0xb4
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Wait.func1(0xc4229f4c00, 0xc420c8ef00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:478 +0x41
created by github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Wait
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:477 +0x8d

goroutine 21215 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x327)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc420c6bda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc420c6bda0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d3e0, 0xc42271a120, 0xc42037d3d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21179 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x312)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc420c6b560)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc420c6b560)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037cab0, 0xc42271a120, 0xc42037caa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21018 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3ce2e80, 0xc4218f3650)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420397b50, 0xc42271a120, 0xc420397b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21302 [select]:
github.com/cockroachdb/cockroach/pkg/kv.GRPCTransportFactory.func1(0x3ce2e00, 0xc420016110)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport_race.go:96 +0x39d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc42271a120, 0x3ce2e00, 0xc420016110, 0xc4204d8020, 0x17, 0x0, 0x0, 0xc4208600a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x14d

goroutine 21316 [runnable]:
internal/race.Disable()
	/usr/local/go/src/internal/race/race.go:28 +0x44
sync.(*RWMutex).RUnlock(0xc420bdaef8)
	/usr/local/go/src/sync/rwmutex.go:66 +0x4d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).GetLease(0xc420bdae00, 0x0, 0x0, 0xc42035fd20, 0x100000001, 0x1, 0xc42035fd30, 0xc42035fd40, 0x0, 0x1, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1206 +0x2cf
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddLocked(0xc422520200, 0x3ce2e80, 0xc421a101e0, 0xc420bdae00, 0x150d5020f9ddc2cd, 0xc400000000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:418 +0x38f
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).MaybeAdd(0xc422520200, 0xc420bdae00, 0x150d5020f9ddc2cd, 0xc400000000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:380 +0xf7
github.com/cockroachdb/cockroach/pkg/storage.(*gcQueue).MaybeAdd(0xc42077a078, 0xc420bdae00, 0x150d5020f9ddc2cd, 0x0)
	<autogenerated>:1 +0x6f
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc4208ba000, 0x3ce2e80, 0xc420e46060, 0xc42271a120, 0x1cda2c69, 0xed1fce34f, 0x0, 0xc420bdae00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:226 +0x3c1
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc420bdae00, 0x58)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:274 +0x145
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc420f7aa50, 0xc4204a0580)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:314 +0x304
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x3ce2e80, 0xc420e46060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:272 +0x3ad
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4211ee410, 0xc42271a120, 0xc4229726c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21225 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x337)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc42223a000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc42223a000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d520, 0xc42271a120, 0xc42037d510)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21241 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x34d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e80f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e80f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d8d0, 0xc42271a120, 0xc42037d8c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21200 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x31d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e80ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e80ae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037cfe0, 0xc42271a120, 0xc42037cf80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21288 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x370)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e81350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e81350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4205000b0, 0xc42271a120, 0xc4205000a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21222 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x34a)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e80d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e80d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d4c0, 0xc42271a120, 0xc42037d4b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21318 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x3ce2e80, 0xc420db82a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:138 +0x911
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4211ee4a0, 0xc42271a120, 0xc4211ee490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21180 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x319)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e808d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e808d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037cae0, 0xc42271a120, 0xc42037cac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 20185 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x2fe)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc420c6ae40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc420c6ae40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037c680, 0xc42271a120, 0xc42037c670)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21263 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x35d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc42223a390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc42223a390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037dc20, 0xc42271a120, 0xc42037dc00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21153 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Unlock(0xc422b54490)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:37 +0x69
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).IsExhausted(0xc422b54460, 0x3ce9900)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:161 +0xbd
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc423c483c0, 0x3ce2e80, 0xc4229326f0, 0xc423c48410, 0x7, 0xc422ab8100, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1242 +0x1dd
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc423c483c0, 0x3ce2e80, 0xc4229326f0, 0x7, 0xc422ab8100, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:382 +0x2f0
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc423c483c0, 0x3ce2e80, 0xc4229326f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42158a4b0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:446 +0x1ab
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc423c483c0, 0x3ce2e80, 0xc4229326f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42158a4b0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1027 +0x403
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc423c483c0, 0x3ce2e80, 0xc4229326f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42158a4b0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:703 +0x1d3d
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc423c483c0, 0x3ce2e80, 0xc4229326f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc422728000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:624 +0x53a
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc421548c60, 0x3ce2e80, 0xc4229326f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc422728000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:434 +0x2bc
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc42137a560, 0x3ce2dc0, 0xc4203a8ac0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc422728000, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:555 +0x2df
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc423791b00, 0x3ce2dc0, 0xc4203a8ac0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:954 +0x691
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send-fm(0x3ce2dc0, 0xc4203a8ac0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:501 +0x97
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x3ce2dc0, 0xc4203a8ac0, 0xc422a8ff48, 0xc423866000, 0xc421700ed8, 0xc422932690)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:459 +0x13e
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc423791b00, 0x3ce2dc0, 0xc4203a8ac0, 0xc423866000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:501 +0x112
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc423791b00, 0x3ce2dc0, 0xc4203a8ac0, 0x28f7680, 0xc4213f1ba0, 0x7f8ec66ead90, 0x0, 0xc422a901c8, 0x1ee8795, 0x7f6db0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:353 +0x115
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).GetProto(0xc423791b00, 0x3ce2dc0, 0xc4203a8ac0, 0x28f7680, 0xc4213f1ba0, 0x3ceb340, 0xc4203b0910, 0x1, 0x8)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:361 +0x85
github.com/cockroachdb/cockroach/pkg/sql.getDescriptorByID(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x1, 0x3cf1a20, 0xc4213f0000, 0xe, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/descriptor.go:196 +0x16d
github.com/cockroachdb/cockroach/pkg/sql.getDescriptor(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cd2880, 0xc421326000, 0x3cf1a20, 0xc4213f0000, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/descriptor.go:180 +0x19f
github.com/cockroachdb/cockroach/pkg/sql.getDatabaseDesc(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cea020, 0xc42137a7c0, 0x2a77350, 0x6, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:125 +0x17e
github.com/cockroachdb/cockroach/pkg/sql.MustGetDatabaseDesc(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cea020, 0xc42137a7c0, 0x2a77350, 0x6, 0xc42043c000, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:150 +0x94
github.com/cockroachdb/cockroach/pkg/sql.getTableOrViewDesc(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cea020, 0xc42137a7c0, 0xc420c48000, 0x48d5a20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:132 +0x197
github.com/cockroachdb/cockroach/pkg/sql.getTableDesc(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cea020, 0xc42137a7c0, 0xc420c48000, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:157 +0x81
github.com/cockroachdb/cockroach/pkg/sql.MustGetTableDesc(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cea020, 0xc42137a7c0, 0xc420c48000, 0x23a5d00, 0x0, 0x0, 0x207c0b2)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:225 +0x92
github.com/cockroachdb/cockroach/pkg/sql.(*TableCollection).getTableVersion(0xc422a441f8, 0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x3cea020, 0xc42137a7c0, 0xc420c48000, 0xc4210528a8, 0x1ed8299, 0xc42137a7c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:340 +0x187
github.com/cockroachdb/cockroach/pkg/sql.(*planner).getTableDesc(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0xc420c48000, 0x1b278b1, 0x0, 0xc420364968)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/data_source.go:604 +0x21e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).getTableScanOrSequenceOrViewPlan(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0xc420c48000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/data_source.go:589 +0x99
github.com/cockroachdb/cockroach/pkg/sql.(*planner).getDataSource(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0x3ccfe40, 0xc420364960, 0x0, 0x0, 0xc420021900, 0xc420748700, 0xc421052b30, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/data_source.go:397 +0x53e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).getDataSource(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0x3ccfb00, 0xc423ed0140, 0x0, 0x0, 0x814f5d, 0x2c, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/data_source.go:445 +0x8b0
github.com/cockroachdb/cockroach/pkg/sql.(*planner).getSources(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0xc420364980, 0x1, 0x1, 0x0, 0x0, 0x0, 0x7f8ec66fa8b8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/data_source.go:258 +0x40e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).initFrom(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0xc422b6a000, 0xc420264cf0, 0x0, 0x0, 0x1b28a01)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/render.go:362 +0x148
github.com/cockroachdb/cockroach/pkg/sql.(*planner).SelectClause(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0xc420264cf0, 0xc42384a0c8, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/render.go:190 +0x189
github.com/cockroachdb/cockroach/pkg/sql.(*planner).Select(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0xc420cc43c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3cecba0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/render.go:123 +0x973
github.com/cockroachdb/cockroach/pkg/sql.(*planner).newPlan(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0x3ce5300, 0xc420cc43c0, 0x0, 0x0, 0x0, 0x1, 0x7, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:615 +0x12d0
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makePlan(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0x3ce5300, 0xc420cc43c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:255 +0x10e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeInternalPlan(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0x2a77338, 0x52, 0xc4218af5c0, 0x2, 0x2, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planner.go:317 +0x1c3
github.com/cockroachdb/cockroach/pkg/sql.(*planner).queryRows(0xc4213b0800, 0x3ce2dc0, 0xc4203a8ac0, 0x2a77338, 0x52, 0xc4218af5c0, 0x2, 0x2, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/planner.go:361 +0x20d
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowsInTransaction(0xc42000e0d0, 0x3ce2dc0, 0xc4203a8ac0, 0x29f9a70, 0x9, 0xc423791b00, 0x2a77338, 0x52, 0xc4218af5c0, 0x2, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:76 +0x20e
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).maybeAdoptJob.func1(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0x10, 0x2852200)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:412 +0x19c
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x3ce2dc0, 0xc4203a8ac0, 0xc423791b00, 0xc4224aece0, 0x0, 0x7fca22)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:510 +0x5f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc423791b00, 0x3ce2dc0, 0xc4203a8ac0, 0xc423790101, 0xc4203b00d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:748 +0x11b
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc42137a560, 0x3ce2dc0, 0xc4203a8ac0, 0xc42042c9e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:509 +0x184
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).maybeAdoptJob(0xc42271a3f0, 0x3ce2e80, 0xc420e46420, 0x3cd0280, 0xc4216ae2a0, 0x980926, 0xc420e46440)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:409 +0x191
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func2(0x3ce2e80, 0xc420e46420)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:207 +0x1a4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421840340, 0xc42271a120, 0xc420e463c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21233 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x341)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e80e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e80e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d680, 0xc42271a120, 0xc42037d650)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21147 [runnable]:
reflect.Value.Type(0x2950b60, 0xc4216ef180, 0x199, 0xb, 0x0)
	/usr/local/go/src/reflect/value.go:1685 +0x399
github.com/cockroachdb/cockroach/pkg/util/protoutil.hookVisitor(0x2950b60, 0xc4216ef180, 0x199, 0x2ae1028)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/randnullability.go:32 +0xe7
github.com/cockroachdb/cockroach/pkg/util/protoutil.RandomZeroInsertingVisitor(0x2950b60, 0xc4216ef180, 0x199)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/randnullability.go:86 +0x59
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x2950b60, 0xc4216ef180, 0x199, 0x2ae1030, 0xc4247f0928, 0x8)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:96 +0x178
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x271e800, 0xc42319a030, 0x197, 0x2ae1030, 0xc4247f0928, 0x7)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:78 +0x440
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x29dc4a0, 0xc42319a000, 0x199, 0x2ae1030, 0xc4247f0928, 0x6)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:86 +0x550
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x29ea940, 0xc42077a0e8, 0x196, 0x2ae1030, 0xc4247f0928, 0x5)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:83 +0x3ad
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x281ac80, 0xc42077a0e8, 0x199, 0x2ae1030, 0xc4247f0928, 0x4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:86 +0x550
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x2808e80, 0xc42077a0e8, 0x16, 0x2ae1030, 0xc4247f0928, 0x3)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:83 +0x3ad
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x281af80, 0xc420352570, 0x194, 0x2ae1030, 0xc4247f0928, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:81 +0x208
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x281aa80, 0xc420352570, 0x199, 0x2ae1030, 0xc4247f0928, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:86 +0x550
github.com/cockroachdb/cockroach/pkg/util/protoutil.visitReplace(0x2953f60, 0xc420352570, 0x16, 0x2ae1030, 0xc4247f0928, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:83 +0x3ad
github.com/cockroachdb/cockroach/pkg/util/protoutil.Walk(0x2953f60, 0xc420352570, 0x2ae1030)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/walk.go:105 +0x13e
github.com/cockroachdb/cockroach/pkg/util/protoutil.MaybeFuzz(0x3ceb340, 0xc420352570, 0x20000000000, 0x8)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/protoutil/marshal.go:53 +0xf1
github.com/cockroachdb/cockroach/pkg/roachpb.(*Value).SetProto(0xc4247f0da0, 0x3ceb340, 0xc420352560, 0x3ceb340, 0xc420352560)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/data.go:379 +0x4d
github.com/cockroachdb/cockroach/pkg/internal/client.marshalValue(0x2953f60, 0xc420352560, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42336a700)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/util.go:90 +0xc59
github.com/cockroachdb/cockroach/pkg/internal/client.(*Batch).put(0xc42336a700, 0x28f7680, 0xc42000ca80, 0x2953f60, 0xc420352560, 0xc42336a700)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/batch.go:367 +0xe9
github.com/cockroachdb/cockroach/pkg/internal/client.(*Batch).Put(0xc42336a700, 0x28f7680, 0xc42000ca80, 0x2953f60, 0xc420352560)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/batch.go:388 +0x66
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Put(0xc4232d3b00, 0x3ce2dc0, 0xc4215c6380, 0x28f7680, 0xc42000ca80, 0x2953f60, 0xc420352560, 0x7f8ec54658f0, 0xc4213d2000)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:374 +0xe3
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).AcquireLease.func1(0x3ce2dc0, 0xc4215c6380, 0xc4232d3b00, 0x10, 0x2852200)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:160 +0x2a9
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x3ce2dc0, 0xc4215c6380, 0xc4232d3b00, 0xc4213d2010, 0x0, 0x7fca22)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:510 +0x5f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc4232d3b00, 0x3ce2dc0, 0xc4215c6380, 0xc4232d0101, 0xc420352020, 0xc422c28380, 0x10)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:748 +0x11b
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc42137a560, 0x3ce2dc0, 0xc4215c6380, 0xc42044ee60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:509 +0x184
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).AcquireLease(0xc421506cf0, 0x3ce2e80, 0xc424568780, 0x18, 0x18, 0xc4204c17c0, 0x7f8ec6597138)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:136 +0x159
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec(0xc421506cf0, 0x3ce2e80, 0xc424568780, 0xc42390a300, 0xc420e99180, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:395 +0xa3
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3ce2e80, 0xc4218f2030)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1098 +0x93d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421840220, 0xc42271a120, 0xc420548480)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21015 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3ce2e80, 0xc421a66570)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420397a90, 0xc42271a120, 0xc420397a70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21261 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x35f)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc42223a360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc42223a360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037dba0, 0xc42271a120, 0xc42037db90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21152 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func1(0x3ce2e80, 0xc4223c2090)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:194 +0x17e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421840330, 0xc42271a120, 0xc420e46390)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21201 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x320)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc420c6bb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc420c6bb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d010, 0xc42271a120, 0xc42037cff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21214 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x326)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e80c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e80c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d3c0, 0xc42271a120, 0xc42037d3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21282 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x36c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e812c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e812c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037dfa0, 0xc42271a120, 0xc42037df70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21295 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3ce2e80, 0xc421e81ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1404 +0x4dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4205001d0, 0xc42271a120, 0xc4244bf590)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21218 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x334)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc421e80c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc421e80c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d440, 0xc42271a120, 0xc42037d430)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21232 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x33f)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc42223a090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc42223a090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037d640, 0xc42271a120, 0xc42037d620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21468 [select]:
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).heartbeatLoop(0xc422058f20, 0x3ce2e80, 0xc423be4f30)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:709 +0x458
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).updateState.func2(0x3ce2e80, 0xc4217b80c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:1028 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc42271a120, 0x3ce2e80, 0xc4217b80c0, 0xc421e2f620, 0x29, 0x3cf1fa0, 0xc42390a3e0, 0xc4203c45a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x14d

goroutine 21271 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x36d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc42223a450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc42223a450)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037de20, 0xc42271a120, 0xc42037de10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21010 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3ce2e80, 0xc422a70030)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203979a0, 0xc42271a120, 0xc420397990)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21281 [semacquire]:
sync.runtime_notifyListWait(0xc422812710, 0x371)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc422812700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4219ba000, 0x3ce2e80, 0xc42223a540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3ce2e80, 0xc42223a540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42037df60, 0xc42271a120, 0xc42037df50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21011 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3ce2e80, 0xc421a66510)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203979d0, 0xc42271a120, 0xc4203979c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 20987 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Wait(0xc4229f4c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:482 +0x162
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc420ba5d40, 0xc4224a9b70, 0xc423655680, 0xc4224a9cc8, 0xc421fbb200, 0xc423c08000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:213 +0xf4e
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).distBackfill.func2(0x3ce2dc0, 0xc420ac7b40, 0xc423655680, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:542 +0x7a2
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x3ce2dc0, 0xc420ac7b40, 0xc423655680, 0xc42429aa98, 0x0, 0x7fca22)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:510 +0x5f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc423655680, 0x3ce2dc0, 0xc420ac7b40, 0xc423650101, 0xc42221ab40, 0x3ce2dc0, 0xc42358f080)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:748 +0x11b
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc42137a560, 0x3ce2dc0, 0xc420ac7b40, 0xc422717200, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:509 +0x184
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).distBackfill(0xc420447a48, 0x3ce2dc0, 0xc42358f080, 0xc423c08000, 0xc42216f790, 0xc400000003, 0x1, 0xc8, 0x2adf710, 0x23, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:479 +0xaba
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).truncateAndBackfillColumns(0xc420447a48, 0x3ce2dc0, 0xc42358f080, 0xc423c08000, 0xc42216f790, 0x3, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:594 +0x9a
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runBackfill(0xc420447a48, 0x3ce2dc0, 0xc42358f080, 0xc42216f790, 0xc423c08000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:186 +0x10fa
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).runStateMachineAndBackfill(0xc420447a48, 0x3ce2dc0, 0xc42358f080, 0xc42216f790, 0xc423c08000, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:717 +0x190
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).exec(0xc420447a48, 0x3ce2dc0, 0xc42358f080, 0x1, 0xc423c08000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:477 +0x528
github.com/cockroachdb/cockroach/pkg/sql.(*schemaChangerCollection).execSchemaChanges(0xc4213f9420, 0x3ce2dc0, 0xc42358f080, 0xc42070fa00, 0xc4213f9300, 0xed1fc0101, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/session.go:1387 +0x6a0
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execParsed(0xc42070fa00, 0xc4213f9300, 0xc423a4c820, 0x1, 0x1, 0x0, 0x0, 0xc4211a0480)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:896 +0x73d
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).execRequest(0xc42070fa00, 0xc4213f9300, 0x2a7a91d, 0x62, 0x0, 0x2a7a91d, 0x62)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:717 +0x55e
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).ExecuteStatements(0xc42070fa00, 0xc4213f9300, 0x2a7a91d, 0x62, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:629 +0x1a4
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).ExecuteStatementsBuffered(0xc42070fa00, 0xc4213f9300, 0x2a7a91d, 0x62, 0x0, 0x1, 0x0, 0xc4213b2e08, 0xc4229b5ee0, 0x145a719, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:577 +0x191
github.com/cockroachdb/cockroach/pkg/sqlmigrations.addRoles(0x3ce2e80, 0xc422a4fc20, 0x3cd4640, 0xc42137a560, 0xc42070fa00, 0xc42106ea70, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:717 +0x11d
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations(0xc422472100, 0x3ce2e80, 0xc422a4fc20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:386 +0xebd
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc42106e580, 0x3ce2e80, 0xc422a4fc20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1236 +0x4e8c
github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start(0xc420b62180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:333 +0x24d
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerRaw(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0x12a
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer(0x3cf36c0, 0xc423c48000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:172 +0x66
github.com/cockroachdb/cockroach/pkg/sql.TestSpanBasedDependencyAnalyzer(0xc423c48000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/parallel_stmts_test.go:346 +0xba
testing.tRunner(0xc423c48000, 0x2adf060)
	/usr/local/go/src/testing/testing.go:746 +0x16d
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x569

goroutine 21199 [semacquire]:
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 26, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/2c85c61e7bc64c7c62aa7e25e9e762180f4e345a

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495069&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/40a7b4e6677987d8235d5b607832c3dbf20a8f2b

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=495726&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/aa73c6491abcee69c9ad3da87ea138e9da0dfbe2

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=503519&tab=buildLog

@petermattis petermattis added this to the 2.0 milestone Feb 21, 2018
@nvanbenschoten
Copy link
Member

Duplicate of #21444.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
3 participants
@nvanbenschoten @petermattis @cockroach-teamcity