Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: tests flaking with "not enough time left on migration lease, terminating for safety" #21444

Closed
cockroach-teamcity opened this issue Jan 14, 2018 · 17 comments
Closed

*: tests flaking with "not enough time left on migration lease, terminating for safety" #21444

cockroach-teamcity opened this issue Jan 14, 2018 · 17 comments
Assignees
@andreimatei
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.0

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/3cd0224615394f7606be514ad42419c538e7bd7b

Parameters:

TAGS=
GOFLAGS=-race

Stress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=478328&tab=buildLog


I180114 09:10:59.204680 7101 server/node.go:433  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I180114 09:10:59.205986 7101 sql/distsql_physical_planner.go:122  [n1] creating DistSQLPlanner with address {tcp 127.0.0.1:38367}
I180114 09:10:59.242933 7101 server/server.go:1164  [n1] starting https server at 127.0.0.1:37309
I180114 09:10:59.250728 7101 server/server.go:1165  [n1] starting grpc/postgres server at 127.0.0.1:38367
I180114 09:10:59.250922 7101 server/server.go:1166  [n1] advertising CockroachDB node at 127.0.0.1:38367
W180114 09:10:59.251407 7101 sql/jobs/registry.go:222  [n1] unable to get node liveness: node not in the liveness table
I180114 09:10:59.278089 19931 storage/replica_command.go:819  [split,n1,s1,r1/1:/M{in-ax}] initiating a split of this range at key /System/"" [r2]
I180114 09:10:59.864511 19931 storage/replica_command.go:819  [split,n1,s1,r2/1:/{System/-Max}] initiating a split of this range at key /System/NodeLiveness [r3]
I180114 09:11:00.041829 7101 sql/event_log.go:115  [n1] Event: "alter_table", target: 12, info: {TableName:eventlog Statement:ALTER TABLE system.eventlog ALTER COLUMN "uniqueID" SET DEFAULT uuid_v4() User:node MutationID:0 CascadeDroppedViews:[]}
I180114 09:11:00.193370 19931 storage/replica_command.go:819  [split,n1,s1,r3/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/NodeLivenessMax [r4]
I180114 09:11:00.215689 7101 sql/lease.go:348  [n1] publish: descID=12 (eventlog) version=2 mtime=2018-01-14 09:11:00.213470356 +0000 UTC
I180114 09:11:00.375225 19931 storage/replica_command.go:819  [split,n1,s1,r4/1:/{System/NodeL…-Max}] initiating a split of this range at key /System/tsd [r5]
I180114 09:11:00.668523 19931 storage/replica_command.go:819  [split,n1,s1,r5/1:/{System/tsd-Max}] initiating a split of this range at key /System/"tse" [r6]
W180114 09:11:00.721658 21638 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "sql txn implicit" id=991a93c3 key=/Table/SystemConfigSpan/Start rw=true pri=0.07748250 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921060.555862335,0 orig=1515921060.555862335,0 max=1515921060.555862335,0 wto=false rop=false seq=7
I180114 09:11:00.756464 7101 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:node}
W180114 09:11:00.785470 21179 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "split" id=32b432d2 key=/Local/Range/System/tsd/RangeDescriptor rw=true pri=0.02501528 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921060.668708973,0 orig=1515921060.668708973,0 max=1515921060.668708973,0 wto=false rop=false seq=2
I180114 09:11:00.900855 19931 storage/replica_command.go:819  [split,n1,s1,r6/1:/{System/tse-Max}] initiating a split of this range at key /Table/SystemConfigSpan/Start [r7]
I180114 09:11:01.090602 19931 storage/replica_command.go:819  [split,n1,s1,r7/1:/{Table/System…-Max}] initiating a split of this range at key /Table/11 [r8]
I180114 09:11:01.320250 19931 storage/replica_command.go:819  [split,n1,s1,r8/1:/{Table/11-Max}] initiating a split of this range at key /Table/12 [r9]
I180114 09:11:01.362293 7101 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:$1 User:node}
W180114 09:11:01.420940 21658 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "split" id=84456346 key=/Local/Range/Table/11/RangeDescriptor rw=true pri=0.05185148 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921061.320475913,0 orig=1515921061.320475913,0 max=1515921061.320475913,0 wto=false rop=false seq=2
I180114 09:11:01.490427 19931 storage/replica_command.go:819  [split,n1,s1,r9/1:/{Table/12-Max}] initiating a split of this range at key /Table/13 [r10]
I180114 09:11:01.608132 7101 sql/event_log.go:115  [n1] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:node}
I180114 09:11:01.788154 19931 storage/replica_command.go:819  [split,n1,s1,r10/1:/{Table/13-Max}] initiating a split of this range at key /Table/14 [r11]
W180114 09:11:01.869457 21722 storage/intent_resolver.go:324  [n1,s1] failed to push during intent resolution: failed to push "split" id=2623bb5a key=/Local/Range/Table/13/RangeDescriptor rw=true pri=0.05554231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921061.788492247,0 orig=1515921061.788492247,0 max=1515921061.788492247,0 wto=false rop=false seq=3
I180114 09:11:01.933590 19931 storage/replica_command.go:819  [split,n1,s1,r11/1:/{Table/14-Max}] initiating a split of this range at key /Table/15 [r12]
W180114 09:12:06.754764 21724 server/server.go:990  The server appears to be unable to contact the other nodes in the cluster. Please try

- starting the other nodes, if you haven't already
- double-checking that the '--join' and '--host' flags are set up correctly
- not using the '--background' flag.

If problems persist, please see https://www.cockroachlabs.com/docs/dev/cluster-setup-troubleshooting.html.
I180114 09:12:14.263542 21587 gossip/gossip.go:487  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180114 09:12:43.987880 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 93.6s
W180114 09:12:50.376704 21692 kv/txn_coord_sender.go:786  [split,s1,r11/1:/{Table/14-Max},n1] heartbeat to "split" id=c572730a key=/Local/Range/Table/14/RangeDescriptor rw=true pri=0.01850347 iso=SERIALIZABLE stat=PENDING epo=0 ts=1515921061.933860211,0 orig=1515921061.933860211,0 max=1515921062.433860211,0 wto=false rop=false seq=3 failed: context deadline exceeded
W180114 09:12:51.943648 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180114 09:13:00.901101 21589 server/status/runtime.go:219  [n1] runtime stats: 783 MiB RSS, 215 goroutines, 16 MiB/57 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (165x)
W180114 09:13:07.569460 19931 internal/client/txn.go:540  [split,n1,s1,r11/1:/{Table/14-Max}] failure aborting transaction: writing transaction timed out or ran on multiple coordinators; abort caused by: context deadline exceeded
W180114 09:13:14.909506 21565 storage/replica.go:1532  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s attempting to acquire lease
W180114 09:13:15.392574 19932 storage/replica.go:1532  [replicate,n1,s1,r6/1:/{System/tse-Table/System…}] have been waiting 1m0s attempting to acquire lease
I180114 09:13:16.494163 21587 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
W180114 09:13:27.245259 21564 storage/replica.go:1532  [n1,s1,r1/1:/{Min-System/}] have been waiting 1m0s attempting to acquire lease
W180114 09:13:27.245259 21566 storage/replica.go:1532  [n1,s1,r3/1:/System/NodeLiveness{-Max}] have been waiting 1m0s attempting to acquire lease
E180114 09:13:37.826736 19931 storage/queue.go:668  [split,n1,s1,r11/1:/{Table/14-Max}] unable to split [n1,s1,r11/1:/{Table/14-Max}] at key "/Table/15": split at key /Table/15 failed: context deadline exceeded
W180114 09:13:50.843243 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 53.2s
W180114 09:13:52.855733 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180114 09:14:02.399390 21587 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180114 09:14:06.727621 21589 server/status/runtime.go:219  [n1] runtime stats: 783 MiB RSS, 216 goroutines, 20 MiB/54 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 0.30cgo/sec, 0.29/0.04 %(u/s)time, 0.04 %gc (1x)
W180114 09:14:06.949166 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 9.9s
W180114 09:14:13.679962 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180114 09:14:26.263723 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 8.9s
W180114 09:14:29.450402 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
I180114 09:14:30.724987 21589 server/status/runtime.go:219  [n1] runtime stats: 783 MiB RSS, 217 goroutines, 15 MiB/58 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 2.34cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
I180114 09:14:30.814952 21245 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921121.444694966,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
W180114 09:14:39.238894 20143 storage/replica.go:2793  [n1,s1,r3/1:/System/NodeLiveness{-Max}] have been waiting 1m0s for proposing command RequestLease [/System/NodeLiveness,/Min)
W180114 09:14:39.238894 21628 storage/replica.go:2793  [n1,s1,r7/1:/Table/{SystemCon…-11}] have been waiting 1m0s for proposing command RequestLease [/Table/SystemConfigSpan/Start,/Min)
W180114 09:14:44.766951 21495 storage/store.go:3791  [n1,s1] handle raft ready: 66.0s [processed=1]
W180114 09:14:48.907784 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 13.6s
W180114 09:14:49.269652 21672 storage/replica.go:1532  [n1,s1,r11/1:/{Table/14-Max}] have been waiting 1m0s attempting to acquire lease
W180114 09:14:49.269728 21734 storage/replica.go:2793  [n1,s1,r1/1:/{Min-System/}] have been waiting 1m0s for proposing command RequestLease [/Min,/Min)
W180114 09:14:49.273370 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180114 09:14:49.486614 21417 storage/replica.go:2805  [n1,s1,r6/1:/{System/tse-Table/System…}] context cancellation after 54.9s of attempting command RequestLease [/System/"tse",/Min)
I180114 09:14:55.923112 21589 server/status/runtime.go:219  [n1] runtime stats: 784 MiB RSS, 217 goroutines, 17 MiB/57 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 0.98cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
W180114 09:14:59.904244 21245 storage/store.go:3791  [n1,s1] handle raft ready: 84.3s [processed=1]
W180114 09:14:59.904250 21507 storage/store.go:3791  [n1,s1] handle raft ready: 68.4s [processed=1]
W180114 09:15:14.360409 21612 storage/node_liveness.go:426  [n1,hb] slow heartbeat took 17.0s
W180114 09:15:20.833392 21612 storage/node_liveness.go:365  [n1,hb] failed node liveness heartbeat: context deadline exceeded
W180114 09:15:21.534283 19088 storage/replica.go:1532  [n1,s1,r4/1:/System/{NodeLive…-tsd}] have been waiting 1m0s attempting to acquire lease
I180114 09:15:22.022870 21587 gossip/gossip.go:487  [n1] gossip status (stalled, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180114 09:15:22.206248 21498 storage/replica_proposal.go:196  [n1,s1,r6/1:/{System/tse-Table/System…}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921225.397078370,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
I180114 09:15:22.206640 21499 storage/replica_proposal.go:196  [n1,s1,r11/1:/{Table/14-Max}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921225.397239416,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
I180114 09:15:22.282658 21589 server/status/runtime.go:219  [n1] runtime stats: 784 MiB RSS, 216 goroutines, 17 MiB/56 MiB/94 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 1.63cgo/sec, 0.01/0.01 %(u/s)time, 0.00 %gc (0x)
I180114 09:15:22.358804 21589 server/status/runtime.go:219  [n1] runtime stats: 765 MiB RSS, 218 goroutines, 18 MiB/30 MiB/68 MiB GO alloc/idle/total, 37 MiB/84 MiB CGO alloc/total, 48.36cgo/sec, 0.32/0.16 %(u/s)time, 0.00 %gc (0x)
I180114 09:15:22.360738 21503 storage/replica_proposal.go:196  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921256.684103499,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
I180114 09:15:22.361370 21508 storage/replica_proposal.go:196  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921271.318173096,0 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=1515921068.164603412,0 pro=1515921059.164736054,0
W180114 09:15:22.457378 21499 storage/store.go:3791  [n1,s1] handle raft ready: 56.9s [processed=1]
W180114 09:15:22.460938 21503 storage/store.go:3791  [n1,s1] handle raft ready: 0.9s [processed=1]
W180114 09:15:22.461478 21498 storage/store.go:3791  [n1,s1] handle raft ready: 56.9s [processed=1]
W180114 09:15:22.478681 21508 storage/store.go:3791  [n1,s1] handle raft ready: 0.8s [processed=1]
W180114 09:15:22.552003 21694 storage/node_liveness.go:426  [n1,s1,r7/1:/Table/{SystemCon…-11}] slow heartbeat took 20.1s
I180114 09:15:22.552287 21612 storage/node_liveness.go:362  [n1,hb] heartbeat failed on epoch increment; retrying
E180114 09:15:22.552804 21707 storage/replica_range_lease.go:256  [n1,s1,r11/1:/{Table/14-Max}] heartbeat failed on epoch increment
E180114 09:15:22.555221 21826 storage/replica_range_lease.go:256  [n1,s1,r4/1:/System/{NodeLive…-tsd}] heartbeat failed on epoch increment
I180114 09:15:22.555687 21536 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921298.639229777,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921121.444694966,0
E180114 09:15:22.563154 21680 storage/replica_range_lease.go:256  [n1,s1,r5/1:/System/ts{d-e}] heartbeat failed on epoch increment
I180114 09:15:22.591414 21545 storage/replica_proposal.go:196  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.556914823,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921256.684103499,0
I180114 09:15:22.610310 21548 storage/replica_proposal.go:196  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.563845401,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921298.639229777,0
I180114 09:15:22.620390 21547 storage/replica_proposal.go:196  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.571172594,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921271.318173096,0
I180114 09:15:22.619674 21546 storage/replica_proposal.go:196  [n1,s1,r11/1:/{Table/14-Max}] new range lease repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921322.554659498,0 following repl=(n1,s1):1 seq=2 start=0.000000000,0 epo=1 pro=1515921225.397239416,0
F180114 09:15:22.665344 19088 sqlmigrations/migrations.go:342  [n1] not enough time left on migration lease, terminating for safety
goroutine 19088 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x47f7101, 0x0, 0x0, 0x1b)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x3e2b640, 0xc400000004, 0x38e0067, 0x1b, 0x156, 0xc4203f2e10, 0x44)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:757 +0x6c8
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3c07c80, 0xc421822210, 0x4, 0x2, 0x0, 0x0, 0xc42064feb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:152 +0x324
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3c07c80, 0xc421822210, 0x1, 0xc400000004, 0x0, 0x0, 0xc42064feb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(0x3c07c80, 0xc421822210, 0xc42064feb8, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:158 +0x88
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations.func2(0x3c07c80, 0xc421822210)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:342 +0x37c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc420276ab0, 0x3c07c80, 0xc421822210, 0xc422810c30, 0x29, 0x0, 0x0, 0xc421676d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:266 +0xf4
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:261 +0x14d

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc4202e6690, 0x2982b9c, 0x1d, 0x2a434b0, 0xc42064fc01)
	/usr/local/go/src/testing/testing.go:790 +0x59b
testing.runTests.func1(0xc4202e6690)
	/usr/local/go/src/testing/testing.go:1004 +0xa8
testing.tRunner(0xc4202e6690, 0xc42064fd50)
	/usr/local/go/src/testing/testing.go:746 +0x16d
testing.runTests(0xc4204e4140, 0x3be06e0, 0xd1, 0xd1, 0xc4201981b0)
	/usr/local/go/src/testing/testing.go:1002 +0x522
testing.(*M).Run(0xc420fc5f20, 0x18)
	/usr/local/go/src/testing/testing.go:921 +0x207
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc42064ff20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:36 +0x15e
main.main()
	github.com/cockroachdb/cockroach/pkg/sql/_test/_testmain.go:464 +0x1ce

goroutine 19 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3e2b640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x8f
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0x13a

goroutine 20 [syscall, 7 minutes]:
os/signal.signal_recv(0x823b11)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 21 [select, 4 minutes, locked to thread]:
runtime.gopark(0x2a46f00, 0x0, 0x295b7d7, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:277 +0xfd
runtime.selectgo(0xc42046b750, 0xc4204c4a80)
	/usr/local/go/src/runtime/select.go:395 +0x12c3
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:511 +0x1a7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1

goroutine 21501 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x147)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08390)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28630, 0xc420276ab0, 0xc422c28620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21447 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x10b)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e325a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e325a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a6260, 0xc420276ab0, 0xc4208a6250)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21527 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x150)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e085d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e085d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28970, 0xc420276ab0, 0xc422c28960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21453 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x114)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a63a0, 0xc420276ab0, 0xc4208a6390)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21460 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x11b)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a6510, 0xc420276ab0, 0xc4208a6500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21276 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x3c07c80, 0xc422c39620)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1160 +0x623
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eefad0, 0xc420276ab0, 0xc420eefac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21458 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x117)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32ab0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a64d0, 0xc420276ab0, 0xc4208a64c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21589 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x3c07c80, 0xc420aec9f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1419 +0x1ab
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c29200, 0xc420276ab0, 0xc4204f8300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21463 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x11c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421512ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421512ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a65d0, 0xc420276ab0, 0xc4208a6580)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21502 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x142)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28650, 0xc420276ab0, 0xc422c28640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21505 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x148)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c286b0, 0xc420276ab0, 0xc422c286a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21519 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x14d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28870, 0xc420276ab0, 0xc422c28860)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21603 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3c07c80, 0xc4213b6b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c29460, 0xc420276ab0, 0xc422c29430)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21476 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x122)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4206620c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4206620c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c282f0, 0xc420276ab0, 0xc422c282c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21532 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x166)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e087e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e087e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28a10, 0xc420276ab0, 0xc422c28a00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21498 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x159)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421512c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421512c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c285d0, 0xc420276ab0, 0xc422c285c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21594 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3c07c80, 0xc420e33050)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c292b0, 0xc420276ab0, 0xc422c292a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21564 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3c07c80, 0xc42039f9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1416 +0x4dd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28e10, 0xc420276ab0, 0xc420d425a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21244 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x128)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42049a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42049a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c281d0, 0xc420276ab0, 0xc422c281c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21574 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials.(*tlsCreds).ClientHandshake(0xc420b10078, 0x3c07c40, 0xc422963b60, 0xc420f48bb0, 0xf, 0x3c139a0, 0xc420b420a8, 0x0, 0x0, 0x45, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/credentials/credentials.go:149 +0x2df
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client(0x3c07c40, 0xc422963b60, 0x3c07bc0, 0xc421c48480, 0xc420f48bb0, 0xf, 0x0, 0x0, 0xc420f48bb0, 0xf, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:182 +0x2902
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.NewClientTransport(0x3c07c40, 0xc422963b60, 0x3c07bc0, 0xc421472c00, 0xc420f48bb0, 0xf, 0x0, 0x0, 0xc420f48bb0, 0xf, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:518 +0xd8
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).createTransport(0xc422432500, 0x8, 0x0, 0xbe8ee5c7424f8769, 0x6620f80490, 0x3e296e0, 0xbe8ee5cc0e0f285a, 0x6a9934a381, 0x3e296e0, 0xc421c48400, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1139 +0x3fa
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc422432500, 0x38, 0xc4203eb000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1100 +0x7b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc422432500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1273 +0x61a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc422432500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x269
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0x186

goroutine 21240 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x136)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420662210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420662210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28150, 0xc420276ab0, 0xc422c28140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21477 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x13c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4200e2060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4200e2060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28310, 0xc420276ab0, 0xc422c28300)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21543 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x171)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28b70, 0xc420276ab0, 0xc422c28b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21551 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x16c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f5c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f5c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28c70, 0xc420276ab0, 0xc422c28c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21520 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x151)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e085a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e085a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28890, 0xc420276ab0, 0xc422c28880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21273 [chan receive]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42140d760, 0x2a461a0, 0xc4216ea2c0, 0x3c13820, 0xc420df20a0)
	<autogenerated>:1 +0x8a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4216ea2c0, 0x3c06e40, 0xc42140d760, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:504 +0x265
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x3c07c80, 0xc421566db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:798 +0x8d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eef8d0, 0xc420276ab0, 0xc42140d880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21491 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x137)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420662240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420662240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c284f0, 0xc420276ab0, 0xc422c284e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21517 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x15c)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28830, 0xc420276ab0, 0xc422c28820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21277 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x3c07c80, 0xc422c39650)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1189 +0x332
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eefaf0, 0xc420276ab0, 0xc420eefae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21508 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x162)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e083f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e083f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28710, 0xc420276ab0, 0xc422c28700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21492 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x132)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28510, 0xc420276ab0, 0xc422c28500)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21503 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x156)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e082d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e082d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28670, 0xc420276ab0, 0xc422c28660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21563 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x3c07c80, 0xc421512d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1282 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28e00, 0xc420276ab0, 0xc4204f8000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 19930 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3c07c80, 0xc42039fb00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0x1f8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ea140, 0xc420276ab0, 0xc42022b8e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21243 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x13a)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4206621e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4206621e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c281b0, 0xc420276ab0, 0xc422c281a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21613 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func1(0x3c07c80, 0xc422c380c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:194 +0x17e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210ba330, 0xc420276ab0, 0xc421379f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 20929 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420c80000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1e7
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x2f2

goroutine 21452 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x113)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a6330, 0xc420276ab0, 0xc4208a6320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21223 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc421ca4cc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110 +0x22f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:96 +0x4d

goroutine 21538 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0xfd)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4200e2090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4200e2090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28ad0, 0xc420276ab0, 0xc422c28ac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21454 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x10d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32600)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a63c0, 0xc420276ab0, 0xc4208a63b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 7105 [runnable]:
github.com/cockroachdb/cockroach/pkg/sql.TestTxnCanStillResolveOldName.func2(0xc420ded000, 0x24, 0x40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rename_test.go:148 +0x163
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x3c07c80, 0xc421566cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1516 +0xd83
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eef550, 0xc420276ab0, 0xc421823890)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21507 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x145)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c286f0, 0xc420276ab0, 0xc422c286e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21478 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x121)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc4200e2030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc4200e2030)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28330, 0xc420276ab0, 0xc422c28320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21533 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x15f)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e089c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e089c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28a30, 0xc420276ab0, 0xc422c28a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21609 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Executor).Start.func1(0x3c07c80, 0xc4213b7620)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/executor.go:417 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c29540, 0xc420276ab0, 0xc4204f8380)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21490 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x134)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421202000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421202000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c284d0, 0xc420276ab0, 0xc422c284c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21611 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x3c07c80, 0xc422522000)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:125 +0x1b6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4210ba160, 0xc420276ab0, 0xc4210ba140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21226 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0xfe)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a60c0, 0xc420276ab0, 0xc4208a60b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21475 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x135)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc421d00000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc421d00000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c282b0, 0xc420276ab0, 0xc422c282a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21449 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc421b30b90, 0x111)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc420e32840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc420e32840)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208a62a0, 0xc420276ab0, 0xc4208a6290)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21534 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x160)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc423e08990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc423e08990)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28a50, 0xc420276ab0, 0xc422c28a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21542 [semacquire]:
sync.runtime_notifyListWait(0xc421b30b90, 0x16d)
	/usr/local/go/src/runtime/sema.go:507 +0x11f
sync.(*Cond).Wait(0xc421b30b80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4208bd710, 0x3c07c80, 0xc42039f5f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3c07c80, 0xc42039f5f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422c28b50, 0xc420276ab0, 0xc422c28b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21275 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x3c07c80, 0xc421566de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:371 +0x83
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420eefab0, 0xc420276ab0, 0xc42140d9e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:195 +0x161
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:188 +0xbb

goroutine 21590 [runnable]:
github.com/cockroachdb/cockroach/pkg/roachpb.Span.Contains(0xc4201f7140, 0x2d, 0x30, 0xc4201f7140, 0x2e, 0x30, 0xc42178dd00, 0x24, 0x40, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/data.go:1289 +0x2ed
github.com/cockroachdb/cockroach/pkg/storage/spanset.(*SpanSet).CheckAllowed(0xc42369daa0, 0x1, 0xc42178dd00, 0x24, 0x40, 0x0, 0x0, 0x0, 0xc4230e7688, 0xc4211bfc90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/spanset/spanset.go:120 +0x248
github.com/cockroachdb/cockroach/pkg/storage/spanset.spanSetWriter.Merge(0x7f1194570ea0, 0xc420c7e000, 0xc42369daa0, 0xc42178dd00, 0x24, 0x40, 0x0, 0x0, 0xc4210cfa00, 0x40, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/spanset/batch.go:278 +0xc3
github.com/cockroachdb/cockroach/pkg/storage/spanset.(*spanSetBatch).Merge(0xc42261bb30, 0xc42178dd00, 0x24, 0x40, 0x0, 0x0, 0xc4210cfa00, 0x40, 0x40, 0xc4211c0a00, ...)
	<autogenerated>:1 +0xdb
github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCMerge(0x3c07c80, 0xc420e58f30, 0x7f11909bf028, 0xc42261bb30, 0xc42165d6c0, 0xc42178dd00, 0x24, 0x40, 0x1509a269432def93, 0x0, ...)
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Jan 14, 2018
@a-robinson a-robinson changed the title sql: TestTxnCanStillResolveOldName failed under stress *: tests flaking with "not enough time left on migration lease, terminating for safety" Jan 16, 2018
This was referenced Jan 16, 2018
Closed
Closed
Closed
@a-robinson
Copy link
Contributor

a-robinson commented Jan 16, 2018 •
edited
Loading

@jordanlewis is there a way for us to know whether a teamcity agent was messed up last night? 4 different tests failed with this error, and they were all running preposterously slow. For example, handle raft ready: 84.3s [processed=1] suggests that the disks were basically unresponsive. I don't believe there's anything to do with the code here, we just have to decide whether the infrastructure issue needs investigation.

@benesch
Copy link
Contributor

benesch commented Jan 16, 2018

It's worth noting that the issues you closed occurred on four separate TeamCity agents. Perhaps GCE was having general trouble?

@a-robinson
Copy link
Contributor

Huh, that changes things a bit. It seems unlikely that the disks in the zone were all busted. If no one has specific ideas, I'd probably wait another couple days and see whether this sort of failure happens again.

@benesch
Copy link
Contributor

benesch commented Jan 16, 2018

That's my best idea too.

This was referenced Jan 17, 2018
Closed
Closed
@a-robinson a-robinson reopened this Jan 17, 2018
@a-robinson
Copy link
Contributor

Well, it happened again last night on TestAmbiguousCommit. Any ideas? I'll try reproducing on a gceworker, but don't have any strong hunches.

@a-robinson
Copy link
Contributor

No luck reproducing in 55 minutes of make stressrace PKG=./pkg/sql TESTS=TestAmbiguousCommit on a gceworker.

@benesch
Copy link
Contributor

benesch commented Jan 18, 2018

One shot in the dark: perhaps another test in the package chews up disk resources. Does it reproduce if you instead stress the whole package?

make stressrace PKG=./pkg/sql

@a-robinson a-robinson mentioned this issue Jan 18, 2018
Closed
@a-robinson
Copy link
Contributor

Still no luck reproducing, not with make stressrace PKG=./pkg/sql or even just make testrace PKG=./pkg/sql. stressrace had a tendency to hit

==570==ERROR: ThreadSanitizer failed to allocate 0x80000 (524288) bytes of DenseSlabAllocator (error code: 12)
FATAL: ThreadSanitizer CHECK failed: ./gotsan.cc:6532 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)

@benesch
Copy link
Contributor

benesch commented Jan 19, 2018

Shoot. Next step is to try to repro on an affected TeamCity agent. Let's kick off a few stress builds via TC during working hours next week. If an agent fails, we can (hopefully) manually repro.

This was referenced Jan 20, 2018
Closed
Closed
Closed
Closed
Closed
@a-robinson
Copy link
Contributor

Looks like this has happened every day since we first saw it, so changes that went in the day before are most suspicious. Both #21398 and #21395 went in that day and are pretty substantial.

@benesch
Copy link
Contributor

benesch commented Jan 22, 2018

I'm stressracing pkg/sql on a preemptible TC agent right now. Hopefully that proves fruitful.

@benesch
Copy link
Contributor

benesch commented Jan 22, 2018

Ok, I think I'm on the right track. make testrace PKG=./pkg/sql consumes 2.5GiB of memory. Either something is going very wrong, or we need to run WAY less of these in parallel during nightly stress. If 16 of these are running at the same time, it's no wonder that some of them start hitting timeouts.

@benesch
Copy link
Contributor

benesch commented Jan 22, 2018

Forgot to mention: the "migration lease expired; terminating for safety" reproduces reliably in about five minutes of stressrace on a preemptible agent.

@a-robinson
Copy link
Contributor

Ah, nice. Is swapping enabled on the machines?

@benesch
Copy link
Contributor

benesch commented Jan 22, 2018 •
edited
Loading

No, it's not. But after about 5m of stressrace running, my SSH connection becomes entirely unresponsive. If I'm lucky, it comes back and tells me that stress found the "migration lease expired" error. So something is throttling aggressively. Perhaps the hypervisor? I'm not sure. I haven't seen an OOM kill for whatever that's worth.

@petermattis petermattis added this to the 2.0 milestone Feb 21, 2018
@nvanbenschoten nvanbenschoten mentioned this issue Feb 26, 2018
Closed
@vivekmenezes
Copy link
Contributor

this looks like it's worth moving to 2.1?

@benesch
Copy link
Contributor

benesch commented Mar 2, 2018

We haven’t seen this in a month, so I’m just going to close it out!

@benesch benesch closed this as completed Mar 2, 2018
@nvanbenschoten nvanbenschoten mentioned this issue Jan 2, 2020
Merged
@nvanbenschoten nvanbenschoten mentioned this issue Mar 30, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreimatei andreimatei

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
6 participants
@andreimatei @benesch @petermattis @a-robinson @vivekmenezes @cockroach-teamcity