Running make with -j24 GOPATH set to /home/ajwerner/go Cleaning old generated files. # The bootstrap target creates, among other things, ./bin/stress. build/builder.sh make bin/.bootstrap Running make with -j24 GOPATH set to /go Cleaning old generated files. Detected change in build system. Rebooting Make. # Special case for 'prefix' variable, which is required by Homebrew. Running make with -j24 GOPATH set to /go Cleaning old generated files. Detected change in build system. Rebooting Make. # Special case for 'prefix' variable, which is required by Homebrew. Running make with -j24 GOPATH set to /go Cleaning old generated files. make: 'bin/.bootstrap' is up to date. build/builder.sh make test GOFLAGS=" -v -c -o partitionccl.test" PKG=./pkg/ccl/partitionccl Running make with -j24 GOPATH set to /go Cleaning old generated files. make[1]: Nothing to be done for 'build_lib_static'. GOFLAGS= go test -v -c -o partitionccl.test -tags ' gss make x86_64_pc_linux_gnu' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v19.2.0-alpha.20190606-1964-g982579c" -X "github.com/cockroachdb/cockroach/pkg/build.rev=982579c72d1925475e93df1a783f48c3911fd2da" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-pc-linux-gnu" ' -run "." -timeout 12m ./pkg/ccl/partitionccl ajwerner-stress: putting (dist) bin.docker_amd64/stress stress . 1: done 2: done 3: done 4: done ajwerner-stress: putting (dist) ./partitionccl.test partitionccl.test .......... 1: done 2: done 3: done 4: done 0 runs so far, 0 failures, over 5s 0 runs so far, 0 failures, over 10s 0 runs so far, 0 failures, over 15s 0 runs so far, 0 failures, over 20s 0 runs so far, 0 failures, over 25s 0 runs so far, 0 failures, over 30s 0 runs so far, 0 failures, over 35s 0 runs so far, 0 failures, over 40s 0 runs so far, 0 failures, over 45s 0 runs so far, 0 failures, over 50s 0 runs so far, 0 failures, over 55s 0 runs so far, 0 failures, over 1m0s 0 runs so far, 0 failures, over 1m5s 0 runs so far, 0 failures, over 1m10s 6 runs so far, 0 failures, over 1m15s 21 runs so far, 0 failures, over 1m20s 48 runs so far, 0 failures, over 1m25s 58 runs so far, 0 failures, over 1m30s 64 runs so far, 0 failures, over 1m35s 64 runs so far, 0 failures, over 1m40s 64 runs so far, 0 failures, over 1m45s 64 runs so far, 0 failures, over 1m50s 64 runs so far, 0 failures, over 1m55s 64 runs so far, 0 failures, over 2m0s 64 runs so far, 0 failures, over 2m5s 64 runs so far, 0 failures, over 2m10s 64 runs so far, 0 failures, over 2m15s 65 runs so far, 0 failures, over 2m20s 69 runs so far, 0 failures, over 2m25s 74 runs so far, 0 failures, over 2m30s 87 runs so far, 0 failures, over 2m35s 103 runs so far, 0 failures, over 2m40s 119 runs so far, 0 failures, over 2m45s 128 runs so far, 0 failures, over 2m50s 128 runs so far, 0 failures, over 2m55s 128 runs so far, 0 failures, over 3m0s 128 runs so far, 0 failures, over 3m5s 128 runs so far, 0 failures, over 3m10s 128 runs so far, 0 failures, over 3m15s 128 runs so far, 0 failures, over 3m20s 128 runs so far, 0 failures, over 3m25s 128 runs so far, 0 failures, over 3m30s 131 runs so far, 0 failures, over 3m35s 134 runs so far, 0 failures, over 3m40s 142 runs so far, 0 failures, over 3m45s 155 runs so far, 0 failures, over 3m50s 165 runs so far, 0 failures, over 3m55s 176 runs so far, 0 failures, over 4m0s 191 runs so far, 0 failures, over 4m5s 192 runs so far, 0 failures, over 4m10s 192 runs so far, 0 failures, over 4m15s 192 runs so far, 0 failures, over 4m20s 192 runs so far, 0 failures, over 4m25s 192 runs so far, 0 failures, over 4m30s 192 runs so far, 0 failures, over 4m35s 192 runs so far, 0 failures, over 4m40s 192 runs so far, 0 failures, over 4m45s 194 runs so far, 0 failures, over 4m50s 201 runs so far, 0 failures, over 4m55s 208 runs so far, 0 failures, over 5m0s 218 runs so far, 0 failures, over 5m5s 231 runs so far, 0 failures, over 5m10s 239 runs so far, 0 failures, over 5m15s 250 runs so far, 0 failures, over 5m20s 252 runs so far, 0 failures, over 5m25s 256 runs so far, 0 failures, over 5m30s 256 runs so far, 0 failures, over 5m35s 256 runs so far, 0 failures, over 5m40s 256 runs so far, 0 failures, over 5m45s 256 runs so far, 0 failures, over 5m50s 256 runs so far, 0 failures, over 5m55s 257 runs so far, 0 failures, over 6m0s 261 runs so far, 0 failures, over 6m5s 267 runs so far, 0 failures, over 6m10s 274 runs so far, 0 failures, over 6m15s 285 runs so far, 0 failures, over 6m20s 291 runs so far, 0 failures, over 6m25s 303 runs so far, 0 failures, over 6m30s 311 runs so far, 0 failures, over 6m35s 314 runs so far, 0 failures, over 6m40s 317 runs so far, 0 failures, over 6m45s 320 runs so far, 0 failures, over 6m50s 320 runs so far, 0 failures, over 6m55s 320 runs so far, 0 failures, over 7m0s 320 runs so far, 0 failures, over 7m5s 321 runs so far, 0 failures, over 7m10s 323 runs so far, 0 failures, over 7m15s 327 runs so far, 0 failures, over 7m20s 333 runs so far, 0 failures, over 7m25s 343 runs so far, 0 failures, over 7m30s 347 runs so far, 0 failures, over 7m35s 355 runs so far, 0 failures, over 7m40s 364 runs so far, 0 failures, over 7m45s 371 runs so far, 0 failures, over 7m50s 376 runs so far, 0 failures, over 7m55s 382 runs so far, 0 failures, over 8m0s 383 runs so far, 0 failures, over 8m5s 384 runs so far, 0 failures, over 8m10s 384 runs so far, 0 failures, over 8m15s 385 runs so far, 0 failures, over 8m20s 388 runs so far, 0 failures, over 8m25s 390 runs so far, 0 failures, over 8m30s 394 runs so far, 0 failures, over 8m35s 400 runs so far, 0 failures, over 8m40s 408 runs so far, 0 failures, over 8m45s 415 runs so far, 0 failures, over 8m50s 421 runs so far, 0 failures, over 8m55s 428 runs so far, 0 failures, over 9m0s 433 runs so far, 0 failures, over 9m5s 439 runs so far, 0 failures, over 9m10s 444 runs so far, 0 failures, over 9m15s 446 runs so far, 0 failures, over 9m20s 449 runs so far, 0 failures, over 9m25s 449 runs so far, 0 failures, over 9m30s 449 runs so far, 0 failures, over 9m35s 452 runs so far, 0 failures, over 9m40s 455 runs so far, 0 failures, over 9m45s 460 runs so far, 0 failures, over 9m50s 462 runs so far, 0 failures, over 9m55s 471 runs so far, 0 failures, over 10m0s 479 runs so far, 0 failures, over 10m5s 486 runs so far, 0 failures, over 10m10s 490 runs so far, 0 failures, over 10m15s 501 runs so far, 0 failures, over 10m20s 502 runs so far, 0 failures, over 10m25s 507 runs so far, 0 failures, over 10m30s 511 runs so far, 0 failures, over 10m35s 513 runs so far, 0 failures, over 10m40s 514 runs so far, 0 failures, over 10m45s 515 runs so far, 0 failures, over 10m50s 516 runs so far, 0 failures, over 10m55s 518 runs so far, 0 failures, over 11m0s 524 runs so far, 0 failures, over 11m5s 530 runs so far, 0 failures, over 11m10s 538 runs so far, 0 failures, over 11m15s 540 runs so far, 0 failures, over 11m20s 552 runs so far, 0 failures, over 11m25s 557 runs so far, 0 failures, over 11m30s 564 runs so far, 0 failures, over 11m35s 569 runs so far, 0 failures, over 11m40s 570 runs so far, 0 failures, over 11m45s 576 runs so far, 0 failures, over 11m50s 577 runs so far, 0 failures, over 11m55s 578 runs so far, 0 failures, over 12m0s 580 runs so far, 0 failures, over 12m5s 581 runs so far, 0 failures, over 12m10s 582 runs so far, 0 failures, over 12m15s 586 runs so far, 0 failures, over 12m20s 594 runs so far, 0 failures, over 12m25s 600 runs so far, 0 failures, over 12m30s 607 runs so far, 0 failures, over 12m35s 616 runs so far, 0 failures, over 12m40s 621 runs so far, 0 failures, over 12m45s 629 runs so far, 0 failures, over 12m50s 632 runs so far, 0 failures, over 12m55s 636 runs so far, 0 failures, over 13m0s 640 runs so far, 0 failures, over 13m5s 641 runs so far, 0 failures, over 13m10s 642 runs so far, 0 failures, over 13m15s 644 runs so far, 0 failures, over 13m20s 646 runs so far, 0 failures, over 13m25s shutting down I190903 18:31:56.647947 1 rand.go:85 Random seed: 670508349572528789 === RUN TestRepartitioning I190903 18:31:56.691831 70 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 18:31:56.691900 70 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node W190903 18:31:56.708598 70 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006" I190903 18:31:56.735275 70 server/server.go:911 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I190903 18:31:56.764206 70 server/config.go:502 [n?] 1 storage engine initialized I190903 18:31:56.764244 70 server/config.go:505 [n?] RocksDB cache size: 128 MiB I190903 18:31:56.764254 70 server/config.go:505 [n?] store 0: in-memory, size 0 B I190903 18:31:56.764375 70 server/server.go:1359 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node. I190903 18:31:56.764410 70 server/server.go:1369 [n?] **** add additional nodes by specifying --join=127.0.0.1:42401 I190903 18:31:56.770774 70 server/node.go:317 [n?] **** cluster 7be57138-5ca5-4531-b57d-41abf90f5d38 has been created I190903 18:31:56.771160 70 gossip/gossip.go:394 [n1] NodeDescriptor set to node_id:1 address: attrs:<> locality: tiers: > ServerVersion: build_tag:"v19.2.0-alpha.20190606-1964-g982579c" started_at:1567535516770961620 cluster_name:"" sql_address: I190903 18:31:56.778088 70 server/node.go:422 [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=8.7 KiB), ranges=20, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=929.00 pMax=7848.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00} I190903 18:31:56.778241 70 storage/stores.go:240 [n1] read 0 node addresses from persistent storage W190903 18:31:56.778634 269 storage/store.go:1546 [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown I190903 18:31:56.784641 70 server/node.go:636 [n1] connecting to gossip network to verify cluster ID... I190903 18:31:56.785497 70 server/node.go:656 [n1] node connected via gossip and verified as part of cluster "7be57138-5ca5-4531-b57d-41abf90f5d38" I190903 18:31:56.787579 70 server/node.go:503 [n1] node=1: started with [n1=] engine(s) and attributes [] I190903 18:31:56.787762 70 server/server.go:1989 [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined I190903 18:31:56.796633 70 server/server.go:1486 [n1] starting https server at 127.0.0.1:37375 (use: 127.0.0.1:37375) I190903 18:31:56.796680 70 server/server.go:1491 [n1] starting postgres server at 127.0.0.1:37289 (use: 127.0.0.1:37289) I190903 18:31:56.796703 70 server/server.go:1493 [n1] starting grpc server at 127.0.0.1:42401 I190903 18:31:56.796735 70 server/server.go:1494 [n1] advertising CockroachDB node at 127.0.0.1:42401 I190903 18:31:56.885421 444 sql/event_log.go:130 [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root} I190903 18:31:56.916111 517 sql/event_log.go:130 [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.1-9 User:root} I190903 18:31:56.925643 524 sql/event_log.go:130 [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root} I190903 18:31:56.946555 783 sql/event_log.go:130 [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:f7ca2d8a-c713-4717-bf12-82334e957fe7 User:root} I190903 18:31:56.974587 879 sql/event_log.go:130 [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root} I190903 18:31:56.999833 980 sql/event_log.go:130 [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root} I190903 18:31:57.022306 70 server/server.go:1562 [n1] done ensuring all necessary migrations have run I190903 18:31:57.022379 70 server/server.go:1800 [n1] serving sql connections I190903 18:31:57.025669 561 sql/event_log.go:130 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:42401 Attrs: Locality:region=test,dc=dc1 ServerVersion:19.1-9 BuildTag:v19.2.0-alpha.20190606-1964-g982579c StartedAt:1567535516770961620 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:37289} ClusterID:7be57138-5ca5-4531-b57d-41abf90f5d38 StartedAt:1567535516770961620 LastUp:1567535516770961620} I190903 18:31:57.027092 559 server/server_update.go:68 [n1] no need to upgrade, cluster already at the newest version I190903 18:31:57.055986 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.range_split.by_load_enabled Value:false User:root} I190903 18:31:57.059825 70 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 18:31:57.059859 70 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node W190903 18:31:57.077206 70 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006" I190903 18:31:57.131595 70 server/server.go:911 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I190903 18:31:57.166873 70 server/config.go:502 [n?] 1 storage engine initialized I190903 18:31:57.166901 70 server/config.go:505 [n?] RocksDB cache size: 128 MiB I190903 18:31:57.166911 70 server/config.go:505 [n?] store 0: in-memory, size 0 B W190903 18:31:57.167013 70 gossip/gossip.go:1517 [n?] no incoming or outgoing connections I190903 18:31:57.167062 70 server/server.go:1359 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node. I190903 18:31:57.183012 1401 gossip/client.go:124 [n?] started gossip client to 127.0.0.1:42401 I190903 18:31:57.183444 1360 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:43131 I190903 18:31:57.184518 70 server/node.go:636 [n?] connecting to gossip network to verify cluster ID... I190903 18:31:57.184558 70 server/node.go:656 [n?] node connected via gossip and verified as part of cluster "7be57138-5ca5-4531-b57d-41abf90f5d38" I190903 18:31:57.185042 1360 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:43131 I190903 18:31:57.215418 70 server/node.go:372 [n?] new node allocated ID 2 I190903 18:31:57.216112 70 gossip/gossip.go:394 [n2] NodeDescriptor set to node_id:2 address: attrs:<> locality: tiers: > ServerVersion: build_tag:"v19.2.0-alpha.20190606-1964-g982579c" started_at:1567535517216010853 cluster_name:"" sql_address: I190903 18:31:57.216231 70 storage/stores.go:240 [n2] read 0 node addresses from persistent storage I190903 18:31:57.216305 70 storage/stores.go:259 [n2] wrote 1 node addresses to persistent storage I190903 18:31:57.217109 73 storage/stores.go:259 [n1] wrote 1 node addresses to persistent storage I190903 18:31:57.218458 70 server/node.go:611 [n2] bootstrapped store [n2,s2] I190903 18:31:57.218605 70 server/node.go:503 [n2] node=2: started with [n2=] engine(s) and attributes [] I190903 18:31:57.218673 70 server/server.go:1989 [n2] Could not start goroutine dumper worker due to: directory to store dumps could not be determined I190903 18:31:57.218746 70 server/server.go:1486 [n2] starting https server at 127.0.0.1:42601 (use: 127.0.0.1:42601) I190903 18:31:57.218785 70 server/server.go:1491 [n2] starting postgres server at 127.0.0.1:42405 (use: 127.0.0.1:42405) I190903 18:31:57.218804 70 server/server.go:1493 [n2] starting grpc server at 127.0.0.1:43131 I190903 18:31:57.218823 70 server/server.go:1494 [n2] advertising CockroachDB node at 127.0.0.1:43131 W190903 18:31:57.225060 70 jobs/registry.go:339 [n2] unable to get node liveness: node not in the liveness table I190903 18:31:57.225774 70 server/server.go:1562 [n2] done ensuring all necessary migrations have run I190903 18:31:57.225815 70 server/server.go:1800 [n2] serving sql connections I190903 18:31:57.256380 1147 server/server_update.go:68 [n2] no need to upgrade, cluster already at the newest version I190903 18:31:57.300448 1858 storage/replica_command.go:1430 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.312757 1149 sql/event_log.go:130 [n2] Event: "node_join", target: 2, info: {Descriptor:{NodeID:2 Address:127.0.0.1:43131 Attrs: Locality:region=test,dc=dc2 ServerVersion:19.1-9 BuildTag:v19.2.0-alpha.20190606-1964-g982579c StartedAt:1567535517216010853 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:42405} ClusterID:7be57138-5ca5-4531-b57d-41abf90f5d38 StartedAt:1567535517216010853 LastUp:1567535517216010853} I190903 18:31:57.312967 1504 sql/event_log.go:130 [n2,client=127.0.0.1:49478,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.range_split.by_load_enabled Value:false User:root} I190903 18:31:57.316027 70 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 18:31:57.316105 70 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 18:31:57.335030 1858 storage/replica_raft.go:291 [n1,s1,r4/1:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:57.336260 1858 storage/store_snapshot.go:995 [n1,replicate,s1,r4/1:/System{/tsd-tse}] sending LEARNER snapshot ea6d4875 at applied index 22 I190903 18:31:57.343014 1858 storage/store_snapshot.go:1038 [n1,replicate,s1,r4/1:/System{/tsd-tse}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 1301, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:31:57.346585 1932 storage/replica_raftstorage.go:793 [n2,s2,r4/2:{-}] applying LEARNER snapshot [id=ea6d4875 index=22] I190903 18:31:57.347831 1932 storage/replica_raftstorage.go:814 [n2,s2,r4/2:/System{/tsd-tse}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ea6d4875 index=22] I190903 18:31:57.349018 1858 storage/replica_command.go:1430 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas (add [(n2,s2):2] remove []): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] W190903 18:31:57.352795 70 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006" I190903 18:31:57.353258 1858 storage/replica_raft.go:291 [n1,s1,r4/1:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:57.354490 1858 storage/replica_command.go:1430 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:57.364961 1858 storage/replica_raft.go:291 [n1,s1,r4/1:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:31:57.365031 1858 storage/replica_raft.go:304 [n1,s1,r4/1:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:31:57.366162 1858 storage/replica_command.go:979 [n1,replicate,s1,r4/1:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n1,s1,r4/1:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:31:57.366244 1858 storage/queue.go:1027 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n1,s1,r4/1:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:31:57.371885 70 server/server.go:911 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I190903 18:31:57.382661 70 server/config.go:502 [n?] 1 storage engine initialized I190903 18:31:57.382683 70 server/config.go:505 [n?] RocksDB cache size: 128 MiB I190903 18:31:57.382692 70 server/config.go:505 [n?] store 0: in-memory, size 0 B W190903 18:31:57.382763 70 gossip/gossip.go:1517 [n?] no incoming or outgoing connections I190903 18:31:57.382831 70 server/server.go:1359 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node. I190903 18:31:57.400131 2045 gossip/client.go:124 [n?] started gossip client to 127.0.0.1:42401 I190903 18:31:57.400596 1992 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:44991 I190903 18:31:57.402182 70 server/node.go:636 [n?] connecting to gossip network to verify cluster ID... I190903 18:31:57.402210 70 server/node.go:656 [n?] node connected via gossip and verified as part of cluster "7be57138-5ca5-4531-b57d-41abf90f5d38" I190903 18:31:57.403511 1992 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:44991 I190903 18:31:57.479854 70 server/node.go:372 [n?] new node allocated ID 3 I190903 18:31:57.479977 70 gossip/gossip.go:394 [n3] NodeDescriptor set to node_id:3 address: attrs:<> locality: tiers: > ServerVersion: build_tag:"v19.2.0-alpha.20190606-1964-g982579c" started_at:1567535517479907257 cluster_name:"" sql_address: I190903 18:31:57.480054 70 storage/stores.go:240 [n3] read 0 node addresses from persistent storage I190903 18:31:57.480095 70 storage/stores.go:259 [n3] wrote 2 node addresses to persistent storage I190903 18:31:57.480801 73 storage/stores.go:259 [n1] wrote 2 node addresses to persistent storage I190903 18:31:57.481380 1282 storage/stores.go:259 [n2] wrote 2 node addresses to persistent storage I190903 18:31:57.482569 70 server/node.go:611 [n3] bootstrapped store [n3,s3] I190903 18:31:57.482726 70 server/node.go:503 [n3] node=3: started with [n3=] engine(s) and attributes [] I190903 18:31:57.482797 70 server/server.go:1989 [n3] Could not start goroutine dumper worker due to: directory to store dumps could not be determined I190903 18:31:57.482866 70 server/server.go:1486 [n3] starting https server at 127.0.0.1:41829 (use: 127.0.0.1:41829) I190903 18:31:57.482901 70 server/server.go:1491 [n3] starting postgres server at 127.0.0.1:34395 (use: 127.0.0.1:34395) I190903 18:31:57.482919 70 server/server.go:1493 [n3] starting grpc server at 127.0.0.1:44991 I190903 18:31:57.482935 70 server/server.go:1494 [n3] advertising CockroachDB node at 127.0.0.1:44991 I190903 18:31:57.491899 70 server/server.go:1562 [n3] done ensuring all necessary migrations have run I190903 18:31:57.491938 70 server/server.go:1800 [n3] serving sql connections I190903 18:31:57.495860 460 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.499765 460 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:57.502639 460 storage/store_snapshot.go:995 [n1,replicate,s1,r20/1:/{Table/24-Max}] sending LEARNER snapshot 61ba56ca at applied index 16 I190903 18:31:57.502768 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r20/1:/{Table/24-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.504518 1888 storage/replica_raftstorage.go:793 [n2,s2,r20/2:{-}] applying LEARNER snapshot [id=61ba56ca index=16] I190903 18:31:57.505691 1888 storage/replica_raftstorage.go:814 [n2,s2,r20/2:/{Table/24-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=61ba56ca index=16] I190903 18:31:57.508123 460 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:57.509756 2411 sql/event_log.go:130 [n3] Event: "node_join", target: 3, info: {Descriptor:{NodeID:3 Address:127.0.0.1:44991 Attrs: Locality:region=test,dc=dc3 ServerVersion:19.1-9 BuildTag:v19.2.0-alpha.20190606-1964-g982579c StartedAt:1567535517479907257 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:34395} ClusterID:7be57138-5ca5-4531-b57d-41abf90f5d38 StartedAt:1567535517479907257 LastUp:1567535517479907257} I190903 18:31:57.525387 2409 server/server_update.go:68 [n3] no need to upgrade, cluster already at the newest version I190903 18:31:57.529370 460 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:57.530277 460 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:57.531686 1824 sql/event_log.go:130 [n3,client=127.0.0.1:49344,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.range_split.by_load_enabled Value:false User:root} I190903 18:31:57.550412 460 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:57.552332 460 storage/store_snapshot.go:995 [n1,replicate,s1,r20/1:/{Table/24-Max}] sending LEARNER snapshot 9771621d at applied index 20 I190903 18:31:57.552524 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r20/1:/{Table/24-Max}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.555098 2477 storage/replica_raftstorage.go:793 [n3,s3,r20/3:{-}] applying LEARNER snapshot [id=9771621d index=20] I190903 18:31:57.557792 2477 storage/replica_raftstorage.go:814 [n3,s3,r20/3:/{Table/24-Max}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=9771621d index=20] I190903 18:31:57.565041 460 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n3,s3):3] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:57.570917 460 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:57.574953 460 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.579339 460 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.581280 460 storage/store_snapshot.go:995 [n1,replicate,s1,r17/1:/Table/2{1-2}] sending LEARNER snapshot 0a9b124d at applied index 22 I190903 18:31:57.581444 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r17/1:/Table/2{1-2}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.582472 2511 storage/replica_raftstorage.go:793 [n3,s3,r17/2:{-}] applying LEARNER snapshot [id=0a9b124d index=22] I190903 18:31:57.584614 2511 storage/replica_raftstorage.go:814 [n3,s3,r17/2:/Table/2{1-2}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=0a9b124d index=22] I190903 18:31:57.586099 460 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n3,s3):2] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.591175 460 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.592909 460 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.604411 460 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.606183 460 storage/store_snapshot.go:995 [n1,replicate,s1,r17/1:/Table/2{1-2}] sending LEARNER snapshot 18941a9e at applied index 26 I190903 18:31:57.606306 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r17/1:/Table/2{1-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.607430 2738 storage/replica_raftstorage.go:793 [n2,s2,r17/3:{-}] applying LEARNER snapshot [id=18941a9e index=26] I190903 18:31:57.608271 2738 storage/replica_raftstorage.go:814 [n2,s2,r17/3:/Table/2{1-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=18941a9e index=26] I190903 18:31:57.609500 460 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:57.634578 460 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:57.636101 460 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.638300 460 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.639245 460 storage/store_snapshot.go:995 [n1,replicate,s1,r16/1:/Table/2{0-1}] sending LEARNER snapshot 4af13c53 at applied index 16 I190903 18:31:57.639366 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r16/1:/Table/2{0-1}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.640545 2837 storage/replica_raftstorage.go:793 [n3,s3,r16/2:{-}] applying LEARNER snapshot [id=4af13c53 index=16] I190903 18:31:57.642919 2837 storage/replica_raftstorage.go:814 [n3,s3,r16/2:/Table/2{0-1}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=4af13c53 index=16] I190903 18:31:57.644152 460 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n3,s3):2] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.657686 460 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.658232 460 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.668806 460 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.671998 460 storage/store_snapshot.go:995 [n1,replicate,s1,r16/1:/Table/2{0-1}] sending LEARNER snapshot 24cc415a at applied index 20 I190903 18:31:57.672204 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r16/1:/Table/2{0-1}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.673337 2876 storage/replica_raftstorage.go:793 [n2,s2,r16/3:{-}] applying LEARNER snapshot [id=24cc415a index=20] I190903 18:31:57.675858 2876 storage/replica_raftstorage.go:814 [n2,s2,r16/3:/Table/2{0-1}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=24cc415a index=20] I190903 18:31:57.679844 460 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n2,s2):3] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:57.682868 460 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:57.684164 460 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.687773 460 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.689060 460 storage/store_snapshot.go:995 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] sending LEARNER snapshot 075cb5ea at applied index 17 I190903 18:31:57.689256 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.690296 2446 storage/replica_raftstorage.go:793 [n3,s3,r5/2:{-}] applying LEARNER snapshot [id=075cb5ea index=17] I190903 18:31:57.698273 2446 storage/replica_raftstorage.go:814 [n3,s3,r5/2:/{Systemtse-Table/System…}] applied LEARNER snapshot [total=7ms ingestion=4@7ms id=075cb5ea index=17] I190903 18:31:57.698919 460 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n3,s3):2] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.704196 460 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.705057 460 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.717846 460 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.722888 460 storage/store_snapshot.go:995 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] sending LEARNER snapshot 8d2788f7 at applied index 21 I190903 18:31:57.723125 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.725321 2984 storage/replica_raftstorage.go:793 [n2,s2,r5/3:{-}] applying LEARNER snapshot [id=8d2788f7 index=21] I190903 18:31:57.730927 2984 storage/replica_raftstorage.go:814 [n2,s2,r5/3:/{Systemtse-Table/System…}] applied LEARNER snapshot [total=6ms ingestion=4@5ms id=8d2788f7 index=21] I190903 18:31:57.732916 460 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n2,s2):3] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:57.736080 460 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:57.738236 460 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.748224 460 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:57.754709 460 storage/store_snapshot.go:995 [n1,replicate,s1,r9/1:/Table/1{3-4}] sending LEARNER snapshot a7879ecd at applied index 56 I190903 18:31:57.755052 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r9/1:/Table/1{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 102, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.756545 3107 storage/replica_raftstorage.go:793 [n2,s2,r9/2:{-}] applying LEARNER snapshot [id=a7879ecd index=56] I190903 18:31:57.759492 3107 storage/replica_raftstorage.go:814 [n2,s2,r9/2:/Table/1{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=a7879ecd index=56] I190903 18:31:57.760133 460 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:57.773168 460 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:57.773834 460 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:57.785485 460 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:57.787528 460 storage/store_snapshot.go:995 [n1,replicate,s1,r9/1:/Table/1{3-4}] sending LEARNER snapshot 5343af4d at applied index 62 I190903 18:31:57.787903 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r9/1:/Table/1{3-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 116, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.789341 3024 storage/replica_raftstorage.go:793 [n3,s3,r9/3:{-}] applying LEARNER snapshot [id=5343af4d index=62] I190903 18:31:57.790711 3024 storage/replica_raftstorage.go:814 [n3,s3,r9/3:/Table/1{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5343af4d index=62] I190903 18:31:57.791384 460 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:57.799826 460 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:57.801326 460 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.803728 460 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.805120 460 storage/store_snapshot.go:995 [n1,replicate,s1,r14/1:/Table/1{8-9}] sending LEARNER snapshot aa4a6df3 at applied index 16 I190903 18:31:57.805259 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r14/1:/Table/1{8-9}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.806429 3168 storage/replica_raftstorage.go:793 [n3,s3,r14/2:{-}] applying LEARNER snapshot [id=aa4a6df3 index=16] I190903 18:31:57.807583 3168 storage/replica_raftstorage.go:814 [n3,s3,r14/2:/Table/1{8-9}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=aa4a6df3 index=16] I190903 18:31:57.808086 460 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n3,s3):2] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.811054 460 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.811670 460 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.821088 460 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.823232 460 storage/store_snapshot.go:995 [n1,replicate,s1,r14/1:/Table/1{8-9}] sending LEARNER snapshot ca3b8753 at applied index 20 I190903 18:31:57.823355 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r14/1:/Table/1{8-9}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.824484 3252 storage/replica_raftstorage.go:793 [n2,s2,r14/3:{-}] applying LEARNER snapshot [id=ca3b8753 index=20] I190903 18:31:57.826128 3252 storage/replica_raftstorage.go:814 [n2,s2,r14/3:/Table/1{8-9}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=ca3b8753 index=20] I190903 18:31:57.826628 460 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n2,s2):3] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:57.831167 460 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:57.833801 460 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.849000 460 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.852707 460 storage/store_snapshot.go:995 [n1,replicate,s1,r13/1:/Table/1{7-8}] sending LEARNER snapshot ccf55819 at applied index 16 I190903 18:31:57.852815 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r13/1:/Table/1{7-8}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.854159 3295 storage/replica_raftstorage.go:793 [n3,s3,r13/2:{-}] applying LEARNER snapshot [id=ccf55819 index=16] I190903 18:31:57.857247 3295 storage/replica_raftstorage.go:814 [n3,s3,r13/2:/Table/1{7-8}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=ccf55819 index=16] I190903 18:31:57.857739 460 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n3,s3):2] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.867389 460 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.870688 460 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.878757 460 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.883367 460 storage/store_snapshot.go:995 [n1,replicate,s1,r13/1:/Table/1{7-8}] sending LEARNER snapshot 7bcc072e at applied index 20 I190903 18:31:57.883508 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r13/1:/Table/1{7-8}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.885176 2534 storage/replica_raftstorage.go:793 [n2,s2,r13/3:{-}] applying LEARNER snapshot [id=7bcc072e index=20] I190903 18:31:57.889738 2534 storage/replica_raftstorage.go:814 [n2,s2,r13/3:/Table/1{7-8}] applied LEARNER snapshot [total=4ms ingestion=4@4ms id=7bcc072e index=20] I190903 18:31:57.892418 460 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n2,s2):3] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:57.894810 460 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:57.896459 460 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.900439 460 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.903172 460 storage/store_snapshot.go:995 [n1,replicate,s1,r18/1:/Table/2{2-3}] sending LEARNER snapshot 9bd357ca at applied index 16 I190903 18:31:57.903302 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r18/1:/Table/2{2-3}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.908463 2536 storage/replica_raftstorage.go:793 [n3,s3,r18/2:{-}] applying LEARNER snapshot [id=9bd357ca index=16] I190903 18:31:57.909851 2536 storage/replica_raftstorage.go:814 [n3,s3,r18/2:/Table/2{2-3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=9bd357ca index=16] I190903 18:31:57.910317 460 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n3,s3):2] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.913050 460 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.913581 460 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.927235 460 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.929177 460 storage/store_snapshot.go:995 [n1,replicate,s1,r18/1:/Table/2{2-3}] sending LEARNER snapshot be5653c8 at applied index 20 I190903 18:31:57.929303 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r18/1:/Table/2{2-3}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.932795 3525 storage/replica_raftstorage.go:793 [n2,s2,r18/3:{-}] applying LEARNER snapshot [id=be5653c8 index=20] I190903 18:31:57.942767 3525 storage/replica_raftstorage.go:814 [n2,s2,r18/3:/Table/2{2-3}] applied LEARNER snapshot [total=7ms ingestion=4@3ms id=be5653c8 index=20] I190903 18:31:57.945059 460 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n2,s2):3] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:57.949151 460 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:57.950611 460 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, next=2, gen=0] I190903 18:31:57.953830 460 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:57.954928 460 storage/store_snapshot.go:995 [n1,replicate,s1,r15/1:/Table/{19-20}] sending LEARNER snapshot 37ca5b43 at applied index 18 I190903 18:31:57.955074 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r15/1:/Table/{19-20}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.956570 3550 storage/replica_raftstorage.go:793 [n3,s3,r15/2:{-}] applying LEARNER snapshot [id=37ca5b43 index=18] I190903 18:31:57.958286 3550 storage/replica_raftstorage.go:814 [n3,s3,r15/2:/Table/{19-20}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=37ca5b43 index=18] I190903 18:31:57.958846 460 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n3,s3):2] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:57.964008 460 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:57.964501 460 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:57.972717 460 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:57.977794 460 storage/store_snapshot.go:995 [n1,replicate,s1,r15/1:/Table/{19-20}] sending LEARNER snapshot 9a75f74c at applied index 22 I190903 18:31:57.978101 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r15/1:/Table/{19-20}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:57.983904 3515 storage/replica_raftstorage.go:793 [n2,s2,r15/3:{-}] applying LEARNER snapshot [id=9a75f74c index=22] I190903 18:31:57.986484 3515 storage/replica_raftstorage.go:814 [n2,s2,r15/3:/Table/{19-20}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=9a75f74c index=22] I190903 18:31:57.987107 460 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n2,s2):3] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:58.000146 460 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:58.001447 460 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.004277 126 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:31:58.004630 460 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.006581 460 storage/store_snapshot.go:995 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] sending LEARNER snapshot 45d171f3 at applied index 44 I190903 18:31:58.007002 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 63, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.008467 3674 storage/replica_raftstorage.go:793 [n2,s2,r6/2:{-}] applying LEARNER snapshot [id=45d171f3 index=44] I190903 18:31:58.010018 3674 storage/replica_raftstorage.go:814 [n2,s2,r6/2:/Table/{SystemCon…-11}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=45d171f3 index=44] I190903 18:31:58.010538 460 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n2,s2):2] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.013768 460 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.014280 460 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.024333 460 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.028654 460 storage/store_snapshot.go:995 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] sending LEARNER snapshot 8d38eec9 at applied index 48 I190903 18:31:58.028948 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 67, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.032765 3804 storage/replica_raftstorage.go:793 [n3,s3,r6/3:{-}] applying LEARNER snapshot [id=8d38eec9 index=48] I190903 18:31:58.037136 3804 storage/replica_raftstorage.go:814 [n3,s3,r6/3:/Table/{SystemCon…-11}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=8d38eec9 index=48] I190903 18:31:58.038320 460 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n3,s3):3] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.042241 460 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.044091 460 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.046672 460 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.047679 460 storage/store_snapshot.go:995 [n1,replicate,s1,r8/1:/Table/1{2-3}] sending LEARNER snapshot 4fea649f at applied index 46 I190903 18:31:58.048292 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r8/1:/Table/1{2-3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 66, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.049789 3745 storage/replica_raftstorage.go:793 [n2,s2,r8/2:{-}] applying LEARNER snapshot [id=4fea649f index=46] I190903 18:31:58.052748 3745 storage/replica_raftstorage.go:814 [n2,s2,r8/2:/Table/1{2-3}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=4fea649f index=46] I190903 18:31:58.054524 460 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.059548 460 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.060077 460 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.068490 460 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.070438 460 storage/store_snapshot.go:995 [n1,replicate,s1,r8/1:/Table/1{2-3}] sending LEARNER snapshot e2bc7c38 at applied index 50 I190903 18:31:58.070876 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r8/1:/Table/1{2-3}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 70, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.072592 3926 storage/replica_raftstorage.go:793 [n3,s3,r8/3:{-}] applying LEARNER snapshot [id=e2bc7c38 index=50] I190903 18:31:58.076712 3926 storage/replica_raftstorage.go:814 [n3,s3,r8/3:/Table/1{2-3}] applied LEARNER snapshot [total=4ms ingestion=4@4ms id=e2bc7c38 index=50] I190903 18:31:58.077503 460 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n3,s3):3] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.089418 460 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.091674 460 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.096721 460 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:58.098064 460 storage/store_snapshot.go:995 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] sending LEARNER snapshot 5ad2b2f6 at applied index 21 I190903 18:31:58.098192 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.099379 3989 storage/replica_raftstorage.go:793 [n3,s3,r2/2:{-}] applying LEARNER snapshot [id=5ad2b2f6 index=21] I190903 18:31:58.123918 3989 storage/replica_raftstorage.go:814 [n3,s3,r2/2:/System/NodeLiveness{-Max}] applied LEARNER snapshot [total=24ms ingestion=4@24ms id=5ad2b2f6 index=21] I190903 18:31:58.124966 460 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:58.132013 460 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:58.132499 460 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:58.141077 460 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:58.142976 460 storage/store_snapshot.go:995 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] sending LEARNER snapshot 603a87d2 at applied index 25 I190903 18:31:58.143161 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.153555 4055 storage/replica_raftstorage.go:793 [n2,s2,r2/3:{-}] applying LEARNER snapshot [id=603a87d2 index=25] I190903 18:31:58.162948 4055 storage/replica_raftstorage.go:814 [n2,s2,r2/3:/System/NodeLiveness{-Max}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=603a87d2 index=25] I190903 18:31:58.163640 460 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n2,s2):3] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:58.166818 460 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:58.169603 460 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.173035 460 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.174011 460 storage/store_snapshot.go:995 [n1,replicate,s1,r12/1:/Table/1{6-7}] sending LEARNER snapshot e439c62f at applied index 16 I190903 18:31:58.174223 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r12/1:/Table/1{6-7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.175412 4260 storage/replica_raftstorage.go:793 [n2,s2,r12/2:{-}] applying LEARNER snapshot [id=e439c62f index=16] I190903 18:31:58.177674 4260 storage/replica_raftstorage.go:814 [n2,s2,r12/2:/Table/1{6-7}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=e439c62f index=16] I190903 18:31:58.178170 460 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.185286 460 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.185848 460 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.195052 460 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.197412 460 storage/store_snapshot.go:995 [n1,replicate,s1,r12/1:/Table/1{6-7}] sending LEARNER snapshot d5d562db at applied index 21 I190903 18:31:58.198199 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r12/1:/Table/1{6-7}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.201081 4371 storage/replica_raftstorage.go:793 [n3,s3,r12/3:{-}] applying LEARNER snapshot [id=d5d562db index=21] I190903 18:31:58.202781 4371 storage/replica_raftstorage.go:814 [n3,s3,r12/3:/Table/1{6-7}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=d5d562db index=21] I190903 18:31:58.207032 460 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n3,s3):3] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.224161 460 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.229501 460 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.234340 460 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:58.235664 460 storage/store_snapshot.go:995 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] sending LEARNER snapshot c3230e4f at applied index 124 I190903 18:31:58.235919 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 84, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.238473 4463 storage/replica_raftstorage.go:793 [n3,s3,r1/2:{-}] applying LEARNER snapshot [id=c3230e4f index=124] I190903 18:31:58.240022 4463 storage/replica_raftstorage.go:814 [n3,s3,r1/2:/{Min-System/NodeL…}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=c3230e4f index=124] I190903 18:31:58.241181 460 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n3,s3):2] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:58.266838 460 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:58.270923 460 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:58.277102 460 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:58.279429 460 storage/store_snapshot.go:995 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] sending LEARNER snapshot 5cdf2e40 at applied index 131 I190903 18:31:58.279763 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 91, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.285774 4407 storage/replica_raftstorage.go:793 [n2,s2,r1/3:{-}] applying LEARNER snapshot [id=5cdf2e40 index=131] I190903 18:31:58.288475 4407 storage/replica_raftstorage.go:814 [n2,s2,r1/3:/{Min-System/NodeL…}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=5cdf2e40 index=131] I190903 18:31:58.290223 460 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n2,s2):3] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:58.296852 460 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:58.302730 460 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.306856 460 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.308322 460 storage/store_snapshot.go:995 [n1,replicate,s1,r11/1:/Table/1{5-6}] sending LEARNER snapshot 3e327fa5 at applied index 16 I190903 18:31:58.308442 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r11/1:/Table/1{5-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.325804 4511 storage/replica_raftstorage.go:793 [n2,s2,r11/2:{-}] applying LEARNER snapshot [id=3e327fa5 index=16] I190903 18:31:58.339676 4511 storage/replica_raftstorage.go:814 [n2,s2,r11/2:/Table/1{5-6}] applied LEARNER snapshot [total=14ms ingestion=4@2ms id=3e327fa5 index=16] I190903 18:31:58.340401 460 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.344078 460 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.344637 460 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.356091 460 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.360494 460 storage/store_snapshot.go:995 [n1,replicate,s1,r11/1:/Table/1{5-6}] sending LEARNER snapshot 92abb963 at applied index 20 I190903 18:31:58.360770 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r11/1:/Table/1{5-6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.363787 4788 storage/replica_raftstorage.go:793 [n3,s3,r11/3:{-}] applying LEARNER snapshot [id=92abb963 index=20] I190903 18:31:58.368665 4788 storage/replica_raftstorage.go:814 [n3,s3,r11/3:/Table/1{5-6}] applied LEARNER snapshot [total=5ms ingestion=4@4ms id=92abb963 index=20] I190903 18:31:58.373053 460 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.385738 460 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.388073 460 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.391775 460 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:31:58.392830 460 storage/store_snapshot.go:995 [n1,replicate,s1,r10/1:/Table/1{4-5}] sending LEARNER snapshot 7f42fc44 at applied index 16 I190903 18:31:58.392959 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r10/1:/Table/1{4-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.394136 4920 storage/replica_raftstorage.go:793 [n3,s3,r10/2:{-}] applying LEARNER snapshot [id=7f42fc44 index=16] I190903 18:31:58.395639 4920 storage/replica_raftstorage.go:814 [n3,s3,r10/2:/Table/1{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=7f42fc44 index=16] I190903 18:31:58.396970 460 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 18:31:58.420140 460 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:31:58.420761 460 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 18:31:58.429886 460 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:31:58.432912 460 storage/store_snapshot.go:995 [n1,replicate,s1,r10/1:/Table/1{4-5}] sending LEARNER snapshot 43e17ac4 at applied index 20 I190903 18:31:58.433078 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r10/1:/Table/1{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.434355 4895 storage/replica_raftstorage.go:793 [n2,s2,r10/3:{-}] applying LEARNER snapshot [id=43e17ac4 index=20] I190903 18:31:58.436666 4895 storage/replica_raftstorage.go:814 [n2,s2,r10/3:/Table/1{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=43e17ac4 index=20] I190903 18:31:58.437698 460 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 18:31:58.446026 460 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 18:31:58.447905 460 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.455418 460 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.482886 460 storage/store_snapshot.go:995 [n1,replicate,s1,r7/1:/Table/1{1-2}] sending LEARNER snapshot 64572fe6 at applied index 19 I190903 18:31:58.483026 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r7/1:/Table/1{1-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 18:31:58.484175 5046 storage/replica_raftstorage.go:793 [n2,s2,r7/2:{-}] applying LEARNER snapshot [id=64572fe6 index=19] I190903 18:31:58.485758 5046 storage/replica_raftstorage.go:814 [n2,s2,r7/2:/Table/1{1-2}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=64572fe6 index=19] I190903 18:31:58.486948 460 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.506534 460 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.508529 460 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.528073 460 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.530525 460 storage/store_snapshot.go:995 [n1,replicate,s1,r7/1:/Table/1{1-2}] sending LEARNER snapshot dae80b22 at applied index 23 I190903 18:31:58.530675 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r7/1:/Table/1{1-2}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.531861 5326 storage/replica_raftstorage.go:793 [n3,s3,r7/3:{-}] applying LEARNER snapshot [id=dae80b22 index=23] I190903 18:31:58.533276 5326 storage/replica_raftstorage.go:814 [n3,s3,r7/3:/Table/1{1-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=dae80b22 index=23] I190903 18:31:58.533774 460 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n3,s3):3] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.537092 460 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.538298 460 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.541545 460 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.542368 460 storage/store_snapshot.go:995 [n1,replicate,s1,r19/1:/Table/2{3-4}] sending LEARNER snapshot 47c0701c at applied index 17 I190903 18:31:58.542496 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r19/1:/Table/2{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.545306 5352 storage/replica_raftstorage.go:793 [n2,s2,r19/2:{-}] applying LEARNER snapshot [id=47c0701c index=17] I190903 18:31:58.547739 5352 storage/replica_raftstorage.go:814 [n2,s2,r19/2:/Table/2{3-4}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=47c0701c index=17] I190903 18:31:58.548234 460 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.551897 460 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.554741 460 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.571661 460 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.574663 460 storage/store_snapshot.go:995 [n1,replicate,s1,r19/1:/Table/2{3-4}] sending LEARNER snapshot 0ae34124 at applied index 21 I190903 18:31:58.574860 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r19/1:/Table/2{3-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.576420 5408 storage/replica_raftstorage.go:793 [n3,s3,r19/3:{-}] applying LEARNER snapshot [id=0ae34124 index=21] I190903 18:31:58.586267 5408 storage/replica_raftstorage.go:814 [n3,s3,r19/3:/Table/2{3-4}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=0ae34124 index=21] I190903 18:31:58.587054 460 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.606214 460 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.608153 460 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, next=2, gen=0] I190903 18:31:58.610941 460 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:31:58.611994 460 storage/store_snapshot.go:995 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] sending LEARNER snapshot 06bb467e at applied index 48 I190903 18:31:58.612492 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 37, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.615288 5606 storage/replica_raftstorage.go:793 [n2,s2,r3/2:{-}] applying LEARNER snapshot [id=06bb467e index=48] I190903 18:31:58.617107 5606 storage/replica_raftstorage.go:814 [n2,s2,r3/2:/System/{NodeLive…-tsd}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=06bb467e index=48] I190903 18:31:58.618175 460 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n2,s2):2] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 18:31:58.626900 460 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:31:58.628118 460 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.641361 460 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.646881 460 storage/store_snapshot.go:995 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] sending LEARNER snapshot 7aae4062 at applied index 52 I190903 18:31:58.647429 460 storage/store_snapshot.go:1038 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 41, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:58.649543 5662 storage/replica_raftstorage.go:793 [n3,s3,r3/3:{-}] applying LEARNER snapshot [id=7aae4062 index=52] I190903 18:31:58.651258 5662 storage/replica_raftstorage.go:814 [n3,s3,r3/3:/System/{NodeLive…-tsd}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=7aae4062 index=52] I190903 18:31:58.653568 460 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n3,s3):3] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 18:31:58.658628 460 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 18:31:58.659573 460 storage/queue.go:1127 [n1,replicate] purgatory is now empty E190903 18:31:58.663899 5756 storage/queue.go:1027 [n2,replicate,s2,r4/2:/System{/tsd-tse}] no removable replicas from range that needs a removal: [1:0, 2*:34] I190903 18:31:58.673917 70 testutils/testcluster/testcluster.go:669 WaitForFullReplication took: 1.141262694s I190903 18:31:58.686390 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 52, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:31:58.692193 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:server.declined_reservation_timeout Value:00:00:00 User:root} I190903 18:31:58.698718 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:server.failed_reservation_timeout Value:00:00:00 User:root} === RUN TestRepartitioning/unpartitioned/unpartitioned I190903 18:31:58.729430 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 52, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[]} I190903 18:31:58.753266 5813 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 18:31:58.784426 1399 gossip/gossip.go:1531 [n2] node has connected to cluster via gossip I190903 18:31:58.784640 1399 storage/stores.go:259 [n2] wrote 2 node addresses to persistent storage I190903 18:31:58.789434 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 53, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:31:58.805296 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 54, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 18:31:58.807924 6830 storage/replica_command.go:284 [n1,split,s1,r20/1:/{Table/24-Max}] initiating a split of this range at key /Table/54 [r21] (zone config) === RUN TestRepartitioning/unpartitioned/single_col_list_partitioning I190903 18:31:58.819955 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=54 (unpartitioned) version=2 mtime=2019-09-03 18:31:58.811012519 +0000 UTC I190903 18:31:58.828448 5813 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:31:58.830355 5813 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, next=3, gen=3] I190903 18:31:58.849819 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 53, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public.unpartitioned]} I190903 18:31:58.853905 5813 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:31:58.854539 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=54 (unpartitioned) version=3 mtime=2019-09-03 18:31:58.854330321 +0000 UTC I190903 18:31:58.855328 5813 storage/store_snapshot.go:995 [n2,replicate,s2,r4/2:/System{/tsd-tse}] sending LEARNER snapshot 3e091821 at applied index 42 I190903 18:31:58.897498 6520 storage/store.go:2593 [n1,replicaGC,s1,r4/1:/System{/tsd-tse}] removing replica r4/1 I190903 18:31:58.898190 6520 storage/replica_destroy.go:152 [n1,replicaGC,s1,r4/1:/System{/tsd-tse}] removed 6 (0+6) keys in 1ms [clear=0ms commit=0ms] I190903 18:31:58.927283 5594 storage/replica_consistency.go:192 [n2,consistencyChecker,s2,r4/2:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:true LastUpdateNanos:1567535518344870306 IntentAge:0 GCBytesAge:0 LiveBytes:-97887 LiveCount:-1941 KeyBytes:-93930 KeyCount:-1941 ValBytes:-3957 ValCount:-1941 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0} I190903 18:31:58.948736 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 55, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:31:58.961257 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 56, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 18:31:58.962196 7749 storage/replica_command.go:284 [n1,split,s1,r21/1:/{Table/54-Max}] initiating a split of this range at key /Table/56 [r22] (zone config) I190903 18:31:58.967059 97 gossip/gossip.go:1531 [n1] node has connected to cluster via gossip I190903 18:31:58.967216 97 storage/stores.go:259 [n1] wrote 2 node addresses to persistent storage I190903 18:31:58.972362 5813 storage/store_snapshot.go:1038 [n2,replicate,s2,r4/2:/System{/tsd-tse}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 1953, log entries: 0, rate-limit: 8.0 MiB/sec, 0.12s I190903 18:31:58.975726 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=56 (single col list partitioning) version=2 mtime=2019-09-03 18:31:58.970995168 +0000 UTC I190903 18:31:58.986297 2334 storage/queue.go:518 [n3,s3,r21/3:/{Table/54-Max}] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:31:58.989805 7838 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=6] I190903 18:31:58.994643 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=56 (single col list partitioning) version=3 mtime=2019-09-03 18:31:58.994482859 +0000 UTC I190903 18:31:59.018687 7838 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n3,s3):3] next=4 E190903 18:31:59.019535 7838 storage/queue.go:1027 [n1,replicate,s1,r21/1:/Table/5{4-6}] no removable replicas from range that needs a removal: [1*:18, 3:17] I190903 18:31:59.025109 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=56 (single col list partitioning) version=4 mtime=2019-09-03 18:31:59.022368261 +0000 UTC I190903 18:31:59.025918 2061 gossip/gossip.go:1531 [n3] node has connected to cluster via gossip I190903 18:31:59.032803 2061 storage/stores.go:259 [n3] wrote 2 node addresses to persistent storage I190903 18:31:59.031268 8243 storage/store.go:2593 [n2,replicaGC,s2,r21/2:/Table/5{4-6}] removing replica r21/2 I190903 18:31:59.033132 8243 storage/replica_destroy.go:152 [n2,replicaGC,s2,r21/2:/Table/5{4-6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.031778 7162 storage/replica_raftstorage.go:793 [n3,s3,r4/3:{-}] applying LEARNER snapshot [id=3e091821 index=42] I190903 18:31:59.038664 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 56, info: {TableName:data.public."single col list partitioning" Statement:ALTER TABLE "single col list partitioning" PARTITION BY LIST (a) (PARTITION p3 VALUES IN ((3)), PARTITION p4 VALUES IN ((4))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:31:59.032136 8340 storage/raft_snapshot_queue.go:120 [n2,raftsnapshot,s2,r4/2:/System{/tsd-tse}] skipping snapshot; replica is likely a learner in the process of being added: (n3,s3):3LEARNER I190903 18:31:59.093729 7162 storage/replica_raftstorage.go:814 [n3,s3,r4/3:/System{/tsd-tse}] applied LEARNER snapshot [total=60ms ingestion=4@60ms id=3e091821 index=42] I190903 18:31:59.094604 5813 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [(n3,s3):3] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=4] I190903 18:31:59.104447 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 56, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:31:59.105508 5813 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:31:59.106736 8681 storage/replica_command.go:284 [n1,split,s1,r22/1:/{Table/56-Max}] initiating a split of this range at key /Table/56/1 [r23] (zone config) I190903 18:31:59.107302 5813 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, (n3,s3):3, next=4, gen=5] I190903 18:31:59.124389 8256 storage/replica_command.go:284 [n1,split,s1,r23/1:/{Table/56/1-Max}] initiating a split of this range at key /Table/56/2 [r24] (zone config) I190903 18:31:59.124669 8765 storage/replica_command.go:1430 [n1,replicate,s1,r22/1:/Table/56{-/1}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r22:/Table/56{-/1} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=7] I190903 18:31:59.131173 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 56, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:31:59.132651 5813 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:31:59.132701 5813 storage/replica_raft.go:304 [n2,s2,r4/2:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:31:59.133616 5813 storage/replica_command.go:979 [n2,replicate,s2,r4/2:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/2:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:31:59.133651 5813 storage/queue.go:1027 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/2:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:31:59.135096 8765 storage/replica_raft.go:291 [n1,s1,r22/1:/Table/56{-/1}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2] next=4 E190903 18:31:59.135789 8060 storage/queue.go:1027 [n3,replicate,s3,r4/3:/System{/tsd-tse}] no removable replicas from range that needs a removal: [2:0, 3*:49] E190903 18:31:59.135879 8765 storage/queue.go:1027 [n1,replicate,s1,r22/1:/Table/56{-/1}] no removable replicas from range that needs a removal: [1*:18, 2:17] I190903 18:31:59.136844 8811 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n3,s3):3, next=4, gen=7] I190903 18:31:59.145067 8600 storage/store.go:2593 [n3,replicaGC,s3,r22/3:/Table/56{-/1}] removing replica r22/3 I190903 18:31:59.145360 9031 storage/replica_command.go:284 [n1,split,s1,r23/1:/Table/56/{1-2}] initiating a split of this range at key /Table/56/1/3 [r25] (zone config) I190903 18:31:59.145457 8600 storage/replica_destroy.go:152 [n3,replicaGC,s3,r22/3:/Table/56{-/1}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.147887 8811 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):1] next=4 I190903 18:31:59.166809 9125 storage/replica_command.go:284 [n1,split,s1,r25/1:/Table/56/{1/3-2}] initiating a split of this range at key /Table/56/1/4 [r26] (zone config) I190903 18:31:59.168333 9140 storage/replica_command.go:1430 [n1,replicate,s1,r23/1:/Table/56/1{-/3}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r23:/Table/56/1{-/3} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=9] I190903 18:31:59.170232 9035 storage/store.go:2593 [n3,replicaGC,s3,r21/3:/Table/5{4-6}] removing replica r21/3 I190903 18:31:59.172259 9035 storage/replica_destroy.go:152 [n3,replicaGC,s3,r21/3:/Table/5{4-6}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.175215 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 56, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:31:59.175402 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.180248 9366 storage/replica_command.go:284 [n1,split,s1,r26/1:/Table/56/{1/4-2}] initiating a split of this range at key /Table/56/1/5 [r27] (zone config) I190903 18:31:59.181231 9140 storage/replica_raft.go:291 [n1,s1,r23/1:/Table/56/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 18:31:59.182384 9140 storage/replica_command.go:1430 [n1,replicate,s1,r23/1:/Table/56/1{-/3}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r23:/Table/56/1{-/3} [(n1,s1):1, (n3,s3):3, next=4, gen=10] I190903 18:31:59.196108 9140 storage/replica_raft.go:291 [n1,s1,r23/1:/Table/56/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):1] next=4 I190903 18:31:59.197800 9412 storage/replica_command.go:1430 [n1,replicate,s1,r24/1:/{Table/56/2-Max}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r24:/{Table/56/2-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=8] I190903 18:31:59.204729 9412 storage/replica_raft.go:291 [n1,s1,r24/1:/{Table/56/2-Max}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 18:31:59.205387 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.205747 9190 storage/store.go:2593 [n2,replicaGC,s2,r23/2:/Table/56/1{-/3}] removing replica r23/2 I190903 18:31:59.206043 9190 storage/replica_destroy.go:152 [n2,replicaGC,s2,r23/2:/Table/56/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.207709 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 18:31:59.207752 9101 storage/queue.go:1027 [n3,replicate,s3,r24/3:/{Table/56/2-Max}] no removable replicas from range that needs a removal: [1:0, 3*:15] I190903 18:31:59.208335 8604 storage/replica_command.go:1430 [n1,replicate,s1,r26/1:/Table/56/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r26:/Table/56/1/{4-5} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=11] I190903 18:31:59.212568 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.221257 8604 storage/replica_raft.go:291 [n1,s1,r26/1:/Table/56/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 18:31:59.221515 9417 storage/store.go:2593 [n3,replicaGC,s3,r23/3:/Table/56/1{-/3}] removing replica r23/3 I190903 18:31:59.221812 9417 storage/replica_destroy.go:152 [n3,replicaGC,s3,r23/3:/Table/56/1{-/3}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.232316 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 E190903 18:31:59.235013 9525 storage/queue.go:1027 [n3,replicate,s3,r26/3:/Table/56/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 3*:20] I190903 18:31:59.289080 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.291066 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.293179 9526 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, (n3,s3):3, next=4, gen=5] I190903 18:31:59.298991 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.301447 9526 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:31:59.301787 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.307465 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:31:59.311617 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:31:59.317784 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.321657 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.328421 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.335481 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.344034 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.346114 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.367968 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.395666 9855 storage/replica_command.go:1430 [n3,replicate,s3,r24/3:/{Table/56/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r24:/{Table/56/2-Max} [(n1,s1):1, (n3,s3):3, next=4, gen=9] I190903 18:31:59.402459 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.435774 9186 storage/store.go:2593 [n2,replicaGC,s2,r24/2:/{Table/56/2-Max}] removing replica r24/2 I190903 18:31:59.436124 9186 storage/replica_destroy.go:152 [n2,replicaGC,s2,r24/2:/{Table/56/2-Max}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.436621 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.436909 9925 storage/store.go:2593 [n2,replicaGC,s2,r26/2:/Table/56/1/{4-5}] removing replica r26/2 I190903 18:31:59.437196 9925 storage/replica_destroy.go:152 [n2,replicaGC,s2,r26/2:/Table/56/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.438560 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.439614 10001 storage/replica_command.go:1430 [n1,replicate,s1,r27/1:/Table/56/{1/5-2}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r27:/Table/56/{1/5-2} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=11] I190903 18:31:59.441756 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.447988 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.449677 9855 storage/replica_raft.go:291 [n3,s3,r24/3:/{Table/56/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 I190903 18:31:59.454147 10001 storage/replica_raft.go:291 [n1,s1,r27/1:/Table/56/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2] next=4 I190903 18:31:59.459094 10001 storage/replica_command.go:1430 [n1,replicate,s1,r27/1:/Table/56/{1/5-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r27:/Table/56/{1/5-2} [(n1,s1):1, (n2,s2):2, next=4, gen=12] I190903 18:31:59.461072 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:31:59.463977 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:31:59.464292 10215 storage/replica_command.go:1430 [n2,replicate,s2,r25/2:/Table/56/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r25:/Table/56/1/{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=10] I190903 18:31:59.480201 10228 storage/store.go:2593 [n1,replicaGC,s1,r24/1:/{Table/56/2-Max}] removing replica r24/1 I190903 18:31:59.481440 10228 storage/replica_destroy.go:152 [n1,replicaGC,s1,r24/1:/{Table/56/2-Max}] removed 6 (0+6) keys in 1ms [clear=1ms commit=0ms] I190903 18:31:59.481719 10001 storage/replica_raft.go:291 [n1,s1,r27/1:/Table/56/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=4 I190903 18:31:59.484481 10215 storage/replica_raft.go:291 [n2,s2,r25/2:/Table/56/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3 (n2,s2):2] next=4 I190903 18:31:59.485518 10215 storage/replica_command.go:1430 [n2,replicate,s2,r25/2:/Table/56/1/{3-4}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r25:/Table/56/1/{3-4} [(n3,s3):3, (n2,s2):2, next=4, gen=11] I190903 18:31:59.487015 10298 storage/store.go:2593 [n2,replicaGC,s2,r4/2:/System{/tsd-tse}] removing replica r4/2 I190903 18:31:59.487804 10298 storage/replica_destroy.go:152 [n2,replicaGC,s2,r4/2:/System{/tsd-tse}] removed 1951 (1941+10) keys in 1ms [clear=1ms commit=0ms] I190903 18:31:59.489858 10034 storage/store.go:2593 [n3,replicaGC,s3,r27/3:/Table/56/{1/5-2}] removing replica r27/3 I190903 18:31:59.490122 10034 storage/replica_destroy.go:152 [n3,replicaGC,s3,r27/3:/Table/56/{1/5-2}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.508282 10215 storage/replica_raft.go:291 [n2,s2,r25/2:/Table/56/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):2] next=4 I190903 18:31:59.538302 10543 storage/store.go:2593 [n2,replicaGC,s2,r27/2:/Table/56/{1/5-2}] removing replica r27/2 I190903 18:31:59.538683 10543 storage/replica_destroy.go:152 [n2,replicaGC,s2,r27/2:/Table/56/{1/5-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.548091 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:31:59.553456 10296 storage/store.go:2593 [n1,replicaGC,s1,r25/1:/Table/56/1/{3-4}] removing replica r25/1 I190903 18:31:59.553830 10499 storage/store.go:2593 [n3,replicaGC,s3,r25/3:/Table/56/1/{3-4}] removing replica r25/3 I190903 18:31:59.554157 10499 storage/replica_destroy.go:152 [n3,replicaGC,s3,r25/3:/Table/56/1/{3-4}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.554228 10296 storage/replica_destroy.go:152 [n1,replicaGC,s1,r25/1:/Table/56/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.560176 6834 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 === RUN TestRepartitioning/unpartitioned/single_col_list_partitioning_-_DEFAULT I190903 18:31:59.572726 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=56 (single col list partitioning) version=5 mtime=2019-09-03 18:31:59.562095831 +0000 UTC E190903 18:31:59.582349 10566 storage/queue.go:1027 [n2,replicate,s2,r22/2:/Table/56{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:21] I190903 18:31:59.595430 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 55, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning"]} I190903 18:31:59.644480 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=56 (single col list partitioning) version=6 mtime=2019-09-03 18:31:59.644316179 +0000 UTC I190903 18:31:59.605510 10919 storage/replica_command.go:1430 [n3,replicate,s3,r26/3:/Table/56/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r26:/Table/56/1/{4-5} [(n1,s1):1, (n3,s3):3, next=4, gen=12] I190903 18:31:59.650339 10747 storage/replica_command.go:1430 [n2,replicate,s2,r22/2:/Table/56{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r22:/Table/56{-/1} [(n1,s1):1, (n2,s2):2, next=4, gen=8] I190903 18:31:59.666978 10747 storage/replica_raft.go:291 [n2,s2,r22/2:/Table/56{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=4 I190903 18:31:59.669284 10919 storage/replica_raft.go:291 [n3,s3,r26/3:/Table/56/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 I190903 18:31:59.678083 11068 storage/store.go:2593 [n1,replicaGC,s1,r22/1:/Table/56{-/1}] removing replica r22/1 I190903 18:31:59.678353 11068 storage/replica_destroy.go:152 [n1,replicaGC,s1,r22/1:/Table/56{-/1}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.713747 11158 storage/store.go:2593 [n1,replicaGC,s1,r26/1:/Table/56/1/{4-5}] removing replica r26/1 I190903 18:31:59.714058 11158 storage/replica_destroy.go:152 [n1,replicaGC,s1,r26/1:/Table/56/1/{4-5}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:31:59.766220 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 57, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:31:59.774990 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 58, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 18:31:59.777952 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=58 (single col list partitioning - DEFAULT) version=2 mtime=2019-09-03 18:31:59.776760069 +0000 UTC I190903 18:31:59.786902 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=58 (single col list partitioning - DEFAULT) version=3 mtime=2019-09-03 18:31:59.786763536 +0000 UTC I190903 18:31:59.791657 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=58 (single col list partitioning - DEFAULT) version=4 mtime=2019-09-03 18:31:59.791047979 +0000 UTC I190903 18:31:59.791833 11774 storage/replica_command.go:284 [n3,split,s3,r24/3:/{Table/56/2-Max}] initiating a split of this range at key /Table/58 [r41] (zone config) I190903 18:31:59.799131 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 58, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:ALTER TABLE "single col list partitioning - DEFAULT" PARTITION BY LIST (a) (PARTITION p4 VALUES IN ((4)), PARTITION p5 VALUES IN ((5)), PARTITION pd VALUES IN ((DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:31:59.836175 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 58, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:31:59.873249 12360 storage/replica_command.go:284 [n3,split,s3,r41/1:/{Table/58-Max}] initiating a split of this range at key /Table/58/1/4 [r42] (zone config) I190903 18:31:59.879508 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 58, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:31:59.899894 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 58, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:31:59.900032 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:31:59.904082 12791 storage/replica_command.go:284 [n3,split,s3,r42/1:/{Table/58/1/4-Max}] initiating a split of this range at key /Table/58/1/5 [r43] (zone config) I190903 18:31:59.910768 12950 storage/replica_command.go:284 [n3,split,s3,r41/1:/Table/58{-/1/4}] initiating a split of this range at key /Table/58/1 [r44] (zone config) I190903 18:31:59.918116 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:31:59.927663 12972 storage/replica_command.go:284 [n3,split,s3,r43/1:/{Table/58/1/5-Max}] initiating a split of this range at key /Table/58/1/6 [r45] (zone config) I190903 18:31:59.927797 12971 storage/replica_command.go:1430 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r42:/Table/58/1/{4-5} [(n3,s3):1, next=2, gen=13] I190903 18:31:59.952314 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:31:59.961074 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:31:59.965096 12971 storage/replica_raft.go:291 [n3,s3,r42/1:/Table/58/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:31:59.966157 12971 storage/store_snapshot.go:995 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] sending LEARNER snapshot bbe81824 at applied index 17 I190903 18:31:59.966277 12971 storage/store_snapshot.go:1038 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:31:59.966470 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:31:59.967892 13206 storage/replica_raftstorage.go:793 [n2,s2,r42/2:{-}] applying LEARNER snapshot [id=bbe81824 index=17] I190903 18:31:59.972516 13206 storage/replica_raftstorage.go:814 [n2,s2,r42/2:/Table/58/1/{4-5}] applied LEARNER snapshot [total=5ms ingestion=4@3ms id=bbe81824 index=17] I190903 18:31:59.973917 12971 storage/replica_command.go:1430 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r42:/Table/58/1/{4-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=14] I190903 18:31:59.982678 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:31:59.997407 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:31:59.997596 13244 storage/replica_command.go:284 [n3,split,s3,r45/1:/{Table/58/1/6-Max}] initiating a split of this range at key /Table/58/2 [r46] (zone config) I190903 18:32:00.002614 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:00.005880 12971 storage/replica_raft.go:291 [n3,s3,r42/1:/Table/58/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:00.006433 12971 storage/replica_command.go:1430 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r42:/Table/58/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=15] I190903 18:32:00.018900 12971 storage/replica_raft.go:291 [n3,s3,r42/1:/Table/58/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:00.018983 12971 storage/replica_raft.go:304 [n3,s3,r42/1:/Table/58/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:00.019798 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.019967 12971 storage/replica_command.go:979 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r42 failed: storage/replica_raft.go:305: [n3,s3,r42/1:/Table/58/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:00.019996 12971 storage/queue.go:1027 [n3,replicate,s3,r42/1:/Table/58/1/{4-5}] change replicas of r42 failed: storage/replica_raft.go:305: [n3,s3,r42/1:/Table/58/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:00.020416 13429 storage/replica_command.go:1430 [n3,replicate,s3,r41/1:/Table/58{-/1}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r41:/Table/58{-/1} [(n3,s3):1, next=2, gen=13] I190903 18:32:00.023657 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.030373 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.039694 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:00.054291 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:00.062373 13429 storage/replica_raft.go:291 [n3,s3,r41/1:/Table/58{-/1}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:00.073084 13429 storage/store_snapshot.go:995 [n3,replicate,s3,r41/1:/Table/58{-/1}] sending LEARNER snapshot 3b7a4111 at applied index 23 I190903 18:32:00.073406 13429 storage/store_snapshot.go:1038 [n3,replicate,s3,r41/1:/Table/58{-/1}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:00.080138 13559 storage/replica_raftstorage.go:793 [n2,s2,r41/2:{-}] applying LEARNER snapshot [id=3b7a4111 index=23] I190903 18:32:00.082611 13559 storage/replica_raftstorage.go:814 [n2,s2,r41/2:/Table/58{-/1}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=3b7a4111 index=23] I190903 18:32:00.092189 13429 storage/replica_command.go:1430 [n3,replicate,s3,r41/1:/Table/58{-/1}] change replicas (add [(n2,s2):2] remove []): existing descriptor r41:/Table/58{-/1} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=14] I190903 18:32:00.098281 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.107563 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.108768 13429 storage/replica_raft.go:291 [n3,s3,r41/1:/Table/58{-/1}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:00.111132 13429 storage/replica_command.go:1430 [n3,replicate,s3,r41/1:/Table/58{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r41:/Table/58{-/1} [(n3,s3):1, (n2,s2):2, next=3, gen=15] I190903 18:32:00.111195 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.130020 13429 storage/replica_raft.go:291 [n3,s3,r41/1:/Table/58{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:00.130070 13429 storage/replica_raft.go:304 [n3,s3,r41/1:/Table/58{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:00.131045 13429 storage/replica_command.go:979 [n3,replicate,s3,r41/1:/Table/58{-/1}] could not promote [] to voter, rolling back: change replicas of r41 failed: storage/replica_raft.go:305: [n3,s3,r41/1:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:00.131086 13429 storage/queue.go:1027 [n3,replicate,s3,r41/1:/Table/58{-/1}] change replicas of r41 failed: storage/replica_raft.go:305: [n3,s3,r41/1:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:00.131548 13858 storage/replica_command.go:1430 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r44:/Table/58/1{-/4} [(n3,s3):1, next=2, gen=13] I190903 18:32:00.136675 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.149956 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.161115 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.175797 13858 storage/replica_raft.go:291 [n3,s3,r44/1:/Table/58/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:00.176805 13858 storage/store_snapshot.go:995 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] sending LEARNER snapshot f1ad989b at applied index 17 I190903 18:32:00.176959 13858 storage/store_snapshot.go:1038 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:00.177900 13678 storage/replica_raftstorage.go:793 [n1,s1,r44/2:{-}] applying LEARNER snapshot [id=f1ad989b index=17] I190903 18:32:00.179241 13678 storage/replica_raftstorage.go:814 [n1,s1,r44/2:/Table/58/1{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=f1ad989b index=17] I190903 18:32:00.180107 13858 storage/replica_command.go:1430 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r44:/Table/58/1{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=14] I190903 18:32:00.196431 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.199713 13858 storage/replica_raft.go:291 [n3,s3,r44/1:/Table/58/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:00.200246 13858 storage/replica_command.go:1430 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r44:/Table/58/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=15] I190903 18:32:00.202476 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.209517 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.216807 13858 storage/replica_raft.go:291 [n3,s3,r44/1:/Table/58/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:00.216845 13858 storage/replica_raft.go:304 [n3,s3,r44/1:/Table/58/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:00.217844 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.218430 13858 storage/replica_command.go:979 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] could not promote [] to voter, rolling back: change replicas of r44 failed: storage/replica_raft.go:305: [n3,s3,r44/1:/Table/58/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:00.218460 13858 storage/queue.go:1027 [n3,replicate,s3,r44/1:/Table/58/1{-/4}] change replicas of r44 failed: storage/replica_raft.go:305: [n3,s3,r44/1:/Table/58/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:00.221579 14172 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [(n2,s2):4LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, next=4, gen=6] E190903 18:32:00.222902 13899 storage/queue.go:1027 [n1,replicate,s1,r44/2:/Table/58/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:23] I190903 18:32:00.243186 14172 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):4LEARNER]: after=[(n3,s3):3 (n2,s2):4LEARNER] next=5 I190903 18:32:00.244867 14172 storage/store_snapshot.go:995 [n3,replicate,s3,r4/3:/System{/tsd-tse}] sending LEARNER snapshot 944030c5 at applied index 57 I190903 18:32:00.248045 14172 storage/store_snapshot.go:1038 [n3,replicate,s3,r4/3:/System{/tsd-tse}] streamed snapshot to (n2,s2):4LEARNER: kv pairs: 1956, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:00.281588 14114 storage/replica_raftstorage.go:793 [n2,s2,r4/4:{-}] applying LEARNER snapshot [id=944030c5 index=57] I190903 18:32:00.288190 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.295507 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:00.300054 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.303355 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.310616 14114 storage/replica_raftstorage.go:814 [n2,s2,r4/4:/System{/tsd-tse}] applied LEARNER snapshot [total=28ms ingestion=4@28ms id=944030c5 index=57] I190903 18:32:00.311310 14172 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [(n2,s2):4] remove []): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, (n2,s2):4LEARNER, next=5, gen=7] I190903 18:32:00.313862 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:00.324981 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.326349 14172 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):4]: after=[(n3,s3):3 (n2,s2):4] next=5 I190903 18:32:00.326971 14172 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, (n2,s2):4, next=5, gen=8] I190903 18:32:00.331781 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.346082 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.364255 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.366417 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.391537 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:00.391719 14172 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 E190903 18:32:00.391861 14172 storage/replica_raft.go:304 [n3,s3,r4/3:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 18:32:00.392319 14724 storage/replica_command.go:1430 [n1,replicate,s1,r44/2:/Table/58/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r44:/Table/58/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=15] I190903 18:32:00.393498 14172 storage/replica_command.go:979 [n3,replicate,s3,r4/3:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n3,s3,r4/3:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) E190903 18:32:00.393538 14172 storage/queue.go:1027 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n3,s3,r4/3:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 18:32:00.396591 14724 storage/replica_raft.go:291 [n1,s1,r44/2:/Table/58/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:00.399580 14521 storage/queue.go:1027 [n2,replicate,s2,r4/4:/System{/tsd-tse}] no removable replicas from range that needs a removal: [3:64, 4*:64] E190903 18:32:00.400126 14002 storage/queue.go:1027 [n2,replicate,s2,r41/2:/Table/58{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 18:32:00.401258 14558 storage/replica_command.go:1430 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r45:/Table/58/{1/6-2} [(n3,s3):1, next=2, gen=15] I190903 18:32:00.405478 14825 storage/store.go:2593 [n3,replicaGC,s3,r44/1:/Table/58/1{-/4}] removing replica r44/1 I190903 18:32:00.405791 14825 storage/replica_destroy.go:152 [n3,replicaGC,s3,r44/1:/Table/58/1{-/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:00.411610 14558 storage/replica_raft.go:291 [n3,s3,r45/1:/Table/58/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:00.413148 14558 storage/store_snapshot.go:995 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] sending LEARNER snapshot db4592f2 at applied index 20 I190903 18:32:00.413389 14558 storage/store_snapshot.go:1038 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:00.415172 14885 storage/replica_raftstorage.go:793 [n1,s1,r45/2:{-}] applying LEARNER snapshot [id=db4592f2 index=20] I190903 18:32:00.416508 14885 storage/replica_raftstorage.go:814 [n1,s1,r45/2:/Table/58/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=db4592f2 index=20] I190903 18:32:00.416960 14558 storage/replica_command.go:1430 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r45:/Table/58/{1/6-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=16] I190903 18:32:00.428115 14558 storage/replica_raft.go:291 [n3,s3,r45/1:/Table/58/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:00.428635 14558 storage/replica_command.go:1430 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r45:/Table/58/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=17] I190903 18:32:00.443353 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:00.444056 14558 storage/replica_raft.go:291 [n3,s3,r45/1:/Table/58/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:00.444103 14558 storage/replica_raft.go:304 [n3,s3,r45/1:/Table/58/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:00.445348 14558 storage/replica_command.go:979 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r45 failed: storage/replica_raft.go:305: [n3,s3,r45/1:/Table/58/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:00.445387 14558 storage/queue.go:1027 [n3,replicate,s3,r45/1:/Table/58/{1/6-2}] change replicas of r45 failed: storage/replica_raft.go:305: [n3,s3,r45/1:/Table/58/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:00.449809 14871 storage/queue.go:1027 [n2,replicate,s2,r42/2:/Table/58/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:00.456705 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:00.530323 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:00.534197 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:00.544633 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:00.548681 10503 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 E190903 18:32:00.558899 15342 storage/queue.go:1027 [n1,replicate,s1,r45/2:/Table/58/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:00.563472 14882 storage/replica_command.go:1430 [n2,replicate,s2,r42/2:/Table/58/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r42:/Table/58/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=15] I190903 18:32:00.576401 14882 storage/replica_raft.go:291 [n2,s2,r42/2:/Table/58/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:00.585959 15456 storage/store.go:2593 [n3,replicaGC,s3,r42/1:/Table/58/1/{4-5}] removing replica r42/1 I190903 18:32:00.586356 15456 storage/replica_destroy.go:152 [n3,replicaGC,s3,r42/1:/Table/58/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] === RUN TestRepartitioning/unpartitioned/single_col_range_partitioning I190903 18:32:00.633802 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=58 (single col list partitioning - DEFAULT) version=5 mtime=2019-09-03 18:32:00.609402503 +0000 UTC I190903 18:32:00.646519 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 57, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning - DEFAULT"]} I190903 18:32:00.648615 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 58 4}] I190903 18:32:00.659534 15892 storage/replica_command.go:1430 [n2,replicate,s2,r41/2:/Table/58{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r41:/Table/58{-/1} [(n3,s3):1, (n2,s2):2, next=3, gen=15] I190903 18:32:00.681339 15892 storage/replica_raft.go:291 [n2,s2,r41/2:/Table/58{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:00.692422 15957 storage/store.go:2593 [n3,replicaGC,s3,r41/1:/Table/58{-/1}] removing replica r41/1 I190903 18:32:00.692841 15957 storage/replica_destroy.go:152 [n3,replicaGC,s3,r41/1:/Table/58{-/1}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:00.704935 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=58 (single col list partitioning - DEFAULT) version=6 mtime=2019-09-03 18:32:00.704732892 +0000 UTC I190903 18:32:00.718797 15991 storage/replica_command.go:1430 [n1,replicate,s1,r45/2:/Table/58/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r45:/Table/58/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=17] I190903 18:32:00.722407 15991 storage/replica_raft.go:291 [n1,s1,r45/2:/Table/58/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:00.727293 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 59, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:00.734438 15953 storage/store.go:2593 [n3,replicaGC,s3,r45/1:/Table/58/{1/6-2}] removing replica r45/1 I190903 18:32:00.734797 15953 storage/replica_destroy.go:152 [n3,replicaGC,s3,r45/1:/Table/58/{1/6-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:00.739251 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:00.740570 15989 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, (n2,s2):4, next=5, gen=8] I190903 18:32:00.746757 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 60, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 18:32:00.748904 16513 storage/replica_command.go:284 [n3,split,s3,r46/1:/{Table/58/2-Max}] initiating a split of this range at key /Table/60 [r47] (zone config) I190903 18:32:00.749080 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=60 (single col range partitioning) version=2 mtime=2019-09-03 18:32:00.747877477 +0000 UTC I190903 18:32:00.756227 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=60 (single col range partitioning) version=3 mtime=2019-09-03 18:32:00.756043896 +0000 UTC I190903 18:32:00.763924 15989 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 I190903 18:32:00.765344 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=60 (single col range partitioning) version=4 mtime=2019-09-03 18:32:00.764663187 +0000 UTC I190903 18:32:00.772416 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 60, info: {TableName:data.public."single col range partitioning" Statement:ALTER TABLE "single col range partitioning" PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:00.816009 16027 storage/store.go:2593 [n3,replicaGC,s3,r4/3:/System{/tsd-tse}] removing replica r4/3 I190903 18:32:00.816405 16027 storage/replica_destroy.go:152 [n3,replicaGC,s3,r4/3:/System{/tsd-tse}] removed 1949 (1941+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:00.843660 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 60, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:00.849694 17476 storage/replica_command.go:284 [n3,split,s3,r47/1:/{Table/60-Max}] initiating a split of this range at key /Table/60/1 [r48] (zone config) I190903 18:32:00.873318 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 60, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:00.887889 16949 storage/replica_command.go:284 [n3,split,s3,r48/1:/{Table/60/1-Max}] initiating a split of this range at key /Table/60/1/3 [r49] (zone config) I190903 18:32:00.891226 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 60, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:00.891750 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:00.912636 18035 storage/replica_command.go:1430 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r48:/Table/60/1{-/3} [(n3,s3):1, next=2, gen=18] I190903 18:32:00.912830 18037 storage/replica_command.go:284 [n3,split,s3,r49/1:/{Table/60/1/3-Max}] initiating a split of this range at key /Table/60/1/4 [r50] (zone config) I190903 18:32:00.919425 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:00.924825 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:00.928209 18084 storage/replica_command.go:284 [n3,split,s3,r50/1:/{Table/60/1/4-Max}] initiating a split of this range at key /Table/60/2 [r51] (zone config) I190903 18:32:00.931958 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:00.934340 18035 storage/replica_raft.go:291 [n3,s3,r48/1:/Table/60/1{-/3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:00.936862 18035 storage/store_snapshot.go:995 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] sending LEARNER snapshot 120afb1a at applied index 17 I190903 18:32:00.937095 18035 storage/store_snapshot.go:1038 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:00.939696 18065 storage/replica_raftstorage.go:793 [n2,s2,r48/2:{-}] applying LEARNER snapshot [id=120afb1a index=17] I190903 18:32:00.942239 18065 storage/replica_raftstorage.go:814 [n2,s2,r48/2:/Table/60/1{-/3}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=120afb1a index=17] I190903 18:32:00.944216 18035 storage/replica_command.go:1430 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r48:/Table/60/1{-/3} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=19] I190903 18:32:00.947814 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:00.955091 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:00.955279 18035 storage/replica_raft.go:291 [n3,s3,r48/1:/Table/60/1{-/3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:00.955830 18035 storage/replica_command.go:1430 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r48:/Table/60/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=20] I190903 18:32:00.971182 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 W190903 18:32:00.972121 18260 storage/intentresolver/intent_resolver.go:508 failed to push during intent resolution: failed to push id=6da8fd28 key=/Local/Range/Table/60/1/RangeDescriptor rw=true pri=0.01117219 stat=PENDING epo=0 ts=1567535520.955676275,0 orig=0.000000000,0 min=1567535520.955676275,0 max=0.000000000,0 wto=false seq=7 I190903 18:32:00.973032 18035 storage/replica_raft.go:291 [n3,s3,r48/1:/Table/60/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:00.973066 18035 storage/replica_raft.go:304 [n3,s3,r48/1:/Table/60/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:00.974234 18035 storage/replica_command.go:979 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] could not promote [] to voter, rolling back: change replicas of r48 failed: storage/replica_raft.go:305: [n3,s3,r48/1:/Table/60/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:00.974314 18035 storage/queue.go:1027 [n3,replicate,s3,r48/1:/Table/60/1{-/3}] change replicas of r48 failed: storage/replica_raft.go:305: [n3,s3,r48/1:/Table/60/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:00.975120 18282 storage/replica_command.go:1430 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r51:/{Table/60/2-Max} [(n3,s3):1, next=2, gen=20] I190903 18:32:00.975773 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:00.981690 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:00.984294 18282 storage/replica_raft.go:291 [n3,s3,r51/1:/{Table/60/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:00.991351 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:00.991511 18282 storage/store_snapshot.go:995 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] sending LEARNER snapshot 889a6735 at applied index 14 I190903 18:32:00.991717 18282 storage/store_snapshot.go:1038 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:00.997842 18452 storage/replica_raftstorage.go:793 [n1,s1,r51/2:{-}] applying LEARNER snapshot [id=889a6735 index=14] I190903 18:32:01.001248 18452 storage/replica_raftstorage.go:814 [n1,s1,r51/2:/{Table/60/2-Max}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=889a6735 index=14] I190903 18:32:01.002026 18282 storage/replica_command.go:1430 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r51:/{Table/60/2-Max} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=21] I190903 18:32:01.003698 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.008130 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.011418 18282 storage/replica_raft.go:291 [n3,s3,r51/1:/{Table/60/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:01.011914 18282 storage/replica_command.go:1430 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r51:/{Table/60/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=22] I190903 18:32:01.012375 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:01.017020 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.024526 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.029495 18282 storage/replica_raft.go:291 [n3,s3,r51/1:/{Table/60/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:01.029577 18282 storage/replica_raft.go:304 [n3,s3,r51/1:/{Table/60/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:01.031710 18282 storage/replica_command.go:979 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] could not promote [] to voter, rolling back: change replicas of r51 failed: storage/replica_raft.go:305: [n3,s3,r51/1:/{Table/60/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:01.031743 18282 storage/queue.go:1027 [n3,replicate,s3,r51/1:/{Table/60/2-Max}] change replicas of r51 failed: storage/replica_raft.go:305: [n3,s3,r51/1:/{Table/60/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:01.033369 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:01.034919 18338 storage/replica_command.go:1430 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r50:/Table/60/{1/4-2} [(n3,s3):1, next=2, gen=20] I190903 18:32:01.047352 18338 storage/replica_raft.go:291 [n3,s3,r50/1:/Table/60/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:01.049045 18338 storage/store_snapshot.go:995 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] sending LEARNER snapshot 559d71e7 at applied index 19 I190903 18:32:01.049256 18338 storage/store_snapshot.go:1038 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:01.049488 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:01.050646 18708 storage/replica_raftstorage.go:793 [n1,s1,r50/2:{-}] applying LEARNER snapshot [id=559d71e7 index=19] I190903 18:32:01.052002 18708 storage/replica_raftstorage.go:814 [n1,s1,r50/2:/Table/60/{1/4-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=559d71e7 index=19] I190903 18:32:01.052516 18338 storage/replica_command.go:1430 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r50:/Table/60/{1/4-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=21] I190903 18:32:01.055144 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.063797 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.067588 18338 storage/replica_raft.go:291 [n3,s3,r50/1:/Table/60/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:01.068266 18338 storage/replica_command.go:1430 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r50:/Table/60/{1/4-2} [(n3,s3):1, (n1,s1):2, next=3, gen=22] I190903 18:32:01.069003 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.078736 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.101334 18338 storage/replica_raft.go:291 [n3,s3,r50/1:/Table/60/{1/4-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:01.101402 18338 storage/replica_raft.go:304 [n3,s3,r50/1:/Table/60/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:01.117552 18338 storage/replica_command.go:979 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r50 failed: storage/replica_raft.go:305: [n3,s3,r50/1:/Table/60/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:01.117583 18338 storage/queue.go:1027 [n3,replicate,s3,r50/1:/Table/60/{1/4-2}] change replicas of r50 failed: storage/replica_raft.go:305: [n3,s3,r50/1:/Table/60/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:01.119788 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 18:32:01.122161 18967 storage/queue.go:1027 [n1,replicate,s1,r50/2:/Table/60/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:01.126064 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 18:32:01.133239 18729 storage/queue.go:1027 [n2,replicate,s2,r48/2:/Table/60/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:01.134650 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:01.194834 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:01.204219 15566 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 18:32:01.210277 19215 storage/queue.go:1027 [n1,replicate,s1,r51/2:/{Table/60/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] === RUN TestRepartitioning/unpartitioned/single_col_range_partitioning_-_MAXVALUE I190903 18:32:01.221874 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=60 (single col range partitioning) version=5 mtime=2019-09-03 18:32:01.211273099 +0000 UTC I190903 18:32:01.227760 19324 storage/replica_command.go:1430 [n2,replicate,s2,r48/2:/Table/60/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r48:/Table/60/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=20] I190903 18:32:01.229937 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 59, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning"]} I190903 18:32:01.232222 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 60 4}] I190903 18:32:01.237697 19324 storage/replica_raft.go:291 [n2,s2,r48/2:/Table/60/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:01.251298 19486 storage/store.go:2593 [n3,replicaGC,s3,r48/1:/Table/60/1{-/3}] removing replica r48/1 I190903 18:32:01.251678 19486 storage/replica_destroy.go:152 [n3,replicaGC,s3,r48/1:/Table/60/1{-/3}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.283760 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=60 (single col range partitioning) version=6 mtime=2019-09-03 18:32:01.283144335 +0000 UTC I190903 18:32:01.315651 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 61, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:01.320805 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 62, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 18:32:01.322125 19185 storage/replica_command.go:284 [n1,split,s1,r51/2:/{Table/60/2-Max}] initiating a split of this range at key /Table/62 [r28] (zone config) I190903 18:32:01.324454 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=62 (single col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 18:32:01.322040807 +0000 UTC I190903 18:32:01.330441 20286 storage/replica_command.go:1430 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r51:/Table/6{0/2-2} [(n3,s3):1, (n1,s1):2, next=3, gen=23] I190903 18:32:01.334260 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=62 (single col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 18:32:01.333507158 +0000 UTC I190903 18:32:01.338241 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:01.341897 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=62 (single col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 18:32:01.341026808 +0000 UTC I190903 18:32:01.347327 20286 storage/replica_raft.go:291 [n1,s1,r51/2:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:01.350016 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 62, info: {TableName:data.public."single col range partitioning - MAXVALUE" Statement:ALTER TABLE "single col range partitioning - MAXVALUE" PARTITION BY RANGE (a) (PARTITION p4 VALUES FROM (minvalue) TO (4), PARTITION p5 VALUES FROM (4) TO (5), PARTITION px VALUES FROM (5) TO (maxvalue)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:01.357866 20616 storage/store.go:2593 [n3,replicaGC,s3,r51/1:/Table/6{0/2-2}] removing replica r51/1 I190903 18:32:01.358237 20616 storage/replica_destroy.go:152 [n3,replicaGC,s3,r51/1:/Table/6{0/2-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.377482 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 62, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:01.378805 20781 storage/replica_command.go:284 [n1,split,s1,r28/2:/{Table/62-Max}] initiating a split of this range at key /Table/62/1 [r29] (zone config) I190903 18:32:01.387104 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 62, info: {Target:PARTITION p5 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:01.387170 21141 storage/replica_command.go:284 [n1,split,s1,r29/2:/{Table/62/1-Max}] initiating a split of this range at key /Table/62/1/4 [r30] (zone config) E190903 18:32:01.390602 20151 storage/queue.go:1027 [n3,replicate,s3,r28/1:/Table/62{-/1}] no removable replicas from range that needs a removal: [1*:18, 2:0] I190903 18:32:01.397133 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 62, info: {Target:PARTITION px OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:01.397247 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.409349 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:01.427502 21607 storage/replica_command.go:284 [n1,split,s1,r30/2:/{Table/62/1/4-Max}] initiating a split of this range at key /Table/62/1/5 [r31] (zone config) I190903 18:32:01.428014 21619 storage/replica_command.go:1430 [n1,replicate,s1,r29/2:/Table/62/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r29:/Table/62/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=25] I190903 18:32:01.430601 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.441708 21619 storage/replica_raft.go:291 [n1,s1,r29/2:/Table/62/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:01.445938 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.447870 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.448559 21641 storage/replica_command.go:284 [n1,split,s1,r31/2:/{Table/62/1/5-Max}] initiating a split of this range at key /Table/62/2 [r32] (zone config) I190903 18:32:01.451002 20210 storage/replica_command.go:1430 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r30:/Table/62/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=26] I190903 18:32:01.456296 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.458332 21717 storage/replica_command.go:1430 [n3,replicate,s3,r46/1:/Table/{58/2-60}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r46:/Table/{58/2-60} [(n3,s3):1, next=2, gen=16] I190903 18:32:01.459230 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.462234 20904 storage/store.go:2593 [n3,replicaGC,s3,r29/1:/Table/62/1{-/4}] removing replica r29/1 I190903 18:32:01.462623 20904 storage/replica_destroy.go:152 [n3,replicaGC,s3,r29/1:/Table/62/1{-/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.463740 20210 storage/replica_raft.go:291 [n1,s1,r30/2:/Table/62/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:01.464853 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.465254 20210 storage/replica_command.go:1430 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r30:/Table/62/1/{4-5} [(n1,s1):2, next=3, gen=27] I190903 18:32:01.487639 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.489126 21689 storage/store.go:2593 [n3,replicaGC,s3,r30/1:/Table/62/1/{4-5}] removing replica r30/1 I190903 18:32:01.489485 21689 storage/replica_destroy.go:152 [n3,replicaGC,s3,r30/1:/Table/62/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.490374 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.493076 21717 storage/replica_raft.go:291 [n3,s3,r46/1:/Table/{58/2-60}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:01.494077 20210 storage/replica_raft.go:291 [n1,s1,r30/2:/Table/62/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:01.495552 20210 storage/store_snapshot.go:995 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] sending LEARNER snapshot 6b3ae6b6 at applied index 20 I190903 18:32:01.495609 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:01.495840 20210 storage/store_snapshot.go:1038 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:01.497333 21150 storage/replica_raftstorage.go:793 [n2,s2,r30/3:{-}] applying LEARNER snapshot [id=6b3ae6b6 index=20] I190903 18:32:01.497644 21717 storage/store_snapshot.go:995 [n3,replicate,s3,r46/1:/Table/{58/2-60}] sending LEARNER snapshot b88b69f3 at applied index 23 I190903 18:32:01.497859 21717 storage/store_snapshot.go:1038 [n3,replicate,s3,r46/1:/Table/{58/2-60}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:01.498421 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:01.499141 21150 storage/replica_raftstorage.go:814 [n2,s2,r30/3:/Table/62/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=6b3ae6b6 index=20] I190903 18:32:01.499701 20210 storage/replica_command.go:1430 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r30:/Table/62/1/{4-5} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=28] I190903 18:32:01.503215 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.506244 20914 storage/replica_raftstorage.go:793 [n2,s2,r46/2:{-}] applying LEARNER snapshot [id=b88b69f3 index=23] I190903 18:32:01.509238 20210 storage/replica_raft.go:291 [n1,s1,r30/2:/Table/62/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:01.511016 20914 storage/replica_raftstorage.go:814 [n2,s2,r46/2:/Table/{58/2-60}] applied LEARNER snapshot [total=5ms ingestion=4@1ms id=b88b69f3 index=23] I190903 18:32:01.511344 20210 storage/replica_command.go:1430 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r30:/Table/62/1/{4-5} [(n1,s1):2, (n2,s2):3, next=4, gen=29] I190903 18:32:01.513242 21717 storage/replica_command.go:1430 [n3,replicate,s3,r46/1:/Table/{58/2-60}] change replicas (add [(n2,s2):2] remove []): existing descriptor r46:/Table/{58/2-60} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=17] I190903 18:32:01.515517 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.518152 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.520782 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.524591 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.525948 21717 storage/replica_raft.go:291 [n3,s3,r46/1:/Table/{58/2-60}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:01.526589 21717 storage/replica_command.go:1430 [n3,replicate,s3,r46/1:/Table/{58/2-60}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r46:/Table/{58/2-60} [(n3,s3):1, (n2,s2):2, next=3, gen=18] I190903 18:32:01.530846 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.533578 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.533877 20210 storage/replica_raft.go:291 [n1,s1,r30/2:/Table/62/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:01.533948 20210 storage/replica_raft.go:304 [n1,s1,r30/2:/Table/62/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:01.535290 20210 storage/replica_command.go:979 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r30 failed: storage/replica_raft.go:305: [n1,s1,r30/2:/Table/62/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:01.535333 20210 storage/queue.go:1027 [n1,replicate,s1,r30/2:/Table/62/1/{4-5}] change replicas of r30 failed: storage/replica_raft.go:305: [n1,s1,r30/2:/Table/62/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:01.535799 21865 storage/replica_command.go:1430 [n1,replicate,s1,r50/2:/Table/60/{1/4-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r50:/Table/60/{1/4-2} [(n3,s3):1, (n1,s1):2, next=3, gen=22] I190903 18:32:01.536872 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.541528 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:01.542157 21865 storage/replica_raft.go:291 [n1,s1,r50/2:/Table/60/{1/4-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:01.544490 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:01.546030 21717 storage/replica_raft.go:291 [n3,s3,r46/1:/Table/{58/2-60}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:01.546105 21717 storage/replica_raft.go:304 [n3,s3,r46/1:/Table/{58/2-60}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:01.547389 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.548480 21717 storage/replica_command.go:979 [n3,replicate,s3,r46/1:/Table/{58/2-60}] could not promote [] to voter, rolling back: change replicas of r46 failed: storage/replica_raft.go:305: [n3,s3,r46/1:/Table/{58/2-60}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:01.548515 21717 storage/queue.go:1027 [n3,replicate,s3,r46/1:/Table/{58/2-60}] change replicas of r46 failed: storage/replica_raft.go:305: [n3,s3,r46/1:/Table/{58/2-60}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:01.549398 22083 storage/queue.go:1027 [n3,replicate,s3,r31/1:/Table/62/{1/5-2}] no removable replicas from range that needs a removal: [1*:18, 2:0] I190903 18:32:01.550583 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.552850 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.555140 21774 storage/store.go:2593 [n3,replicaGC,s3,r50/1:/Table/60/{1/4-2}] removing replica r50/1 I190903 18:32:01.555478 21774 storage/replica_destroy.go:152 [n3,replicaGC,s3,r50/1:/Table/60/{1/4-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.560245 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.562549 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.565904 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.568069 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.571874 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.573915 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 18:32:01.579395 22065 storage/queue.go:1027 [n1,replicate,s1,r32/2:/{Table/62/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:11] I190903 18:32:01.580073 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.582380 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.585658 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.590632 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.593108 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.596294 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.601424 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.603618 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.618413 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.625295 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.635617 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.644055 22321 storage/replica_command.go:1430 [n3,replicate,s3,r31/1:/Table/62/{1/5-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r31:/Table/62/{1/5-2} [(n3,s3):1, (n1,s1):2, next=3, gen=27] I190903 18:32:01.656229 22321 storage/replica_raft.go:291 [n3,s3,r31/1:/Table/62/{1/5-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 18:32:01.664463 21991 storage/store.go:2593 [n1,replicaGC,s1,r31/2:/Table/62/{1/5-2}] removing replica r31/2 I190903 18:32:01.664788 21991 storage/replica_destroy.go:152 [n1,replicaGC,s1,r31/2:/Table/62/{1/5-2}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.667716 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.698167 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.702978 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.739617 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 E190903 18:32:01.763421 22626 storage/queue.go:1027 [n2,replicate,s2,r30/3:/Table/62/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:27] I190903 18:32:01.802381 22774 storage/replica_command.go:1430 [n3,replicate,s3,r28/1:/Table/62{-/1}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r28:/Table/62{-/1} [(n3,s3):1, (n1,s1):2, next=3, gen=24] I190903 18:32:01.809014 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:01.810970 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:01.811358 22774 storage/replica_raft.go:291 [n3,s3,r28/1:/Table/62{-/1}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 E190903 18:32:01.814274 22784 storage/queue.go:1027 [n3,replicate,s3,r32/1:/{Table/62/2-Max}] no removable replicas from range that needs a removal: [1*:14, 2:0] I190903 18:32:01.823083 22558 storage/store.go:2593 [n1,replicaGC,s1,r28/2:/Table/62{-/1}] removing replica r28/2 I190903 18:32:01.823421 22558 storage/replica_destroy.go:152 [n1,replicaGC,s1,r28/2:/Table/62{-/1}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:01.870738 22881 storage/replica_command.go:1430 [n2,replicate,s2,r30/3:/Table/62/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r30:/Table/62/1/{4-5} [(n1,s1):2, (n2,s2):3, next=4, gen=29] I190903 18:32:01.882473 22881 storage/replica_raft.go:291 [n2,s2,r30/3:/Table/62/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:01.890638 23037 storage/store.go:2593 [n1,replicaGC,s1,r30/2:/Table/62/1/{4-5}] removing replica r30/2 I190903 18:32:01.891147 23037 storage/replica_destroy.go:152 [n1,replicaGC,s1,r30/2:/Table/62/1/{4-5}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:01.898154 22994 storage/queue.go:1027 [n2,replicate,s2,r46/2:/Table/{58/2-60}] no removable replicas from range that needs a removal: [1:0, 2*:32] I190903 18:32:01.928286 23146 storage/replica_command.go:1430 [n3,replicate,s3,r32/1:/{Table/62/2-Max}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r32:/{Table/62/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=27] I190903 18:32:01.946131 23146 storage/replica_raft.go:291 [n3,s3,r32/1:/{Table/62/2-Max}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 18:32:01.956746 23084 storage/store.go:2593 [n1,replicaGC,s1,r32/2:/{Table/62/2-Max}] removing replica r32/2 I190903 18:32:01.956998 23084 storage/replica_destroy.go:152 [n1,replicaGC,s1,r32/2:/{Table/62/2-Max}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:02.006272 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:02.011600 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:02.025811 19216 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 === RUN TestRepartitioning/single_col_list_partitioning/single_col_list_partitioning_-_DEFAULT I190903 18:32:02.054221 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=62 (single col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 18:32:02.034878353 +0000 UTC I190903 18:32:02.066364 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 61, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning - MAXVALUE"]} I190903 18:32:02.068589 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 62 4}] I190903 18:32:02.121369 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=62 (single col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 18:32:02.121170568 +0000 UTC I190903 18:32:02.146174 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 63, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:02.146924 23378 storage/replica_command.go:1430 [n2,replicate,s2,r46/2:/Table/{58/2-60}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r46:/Table/{58/2-60} [(n3,s3):1, (n2,s2):2, next=3, gen=18] I190903 18:32:02.152957 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 64, info: {TableName:data.public."single col list partitioning" Statement:CREATE TABLE "single col list partitioning" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p3 VALUES IN (3), PARTITION p4 VALUES IN (4)) User:root} I190903 18:32:02.173799 23378 storage/replica_raft.go:291 [n2,s2,r46/2:/Table/{58/2-60}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:02.175394 24543 storage/replica_command.go:284 [n3,split,s3,r32/1:/{Table/62/2-Max}] initiating a split of this range at key /Table/64 [r52] (zone config) I190903 18:32:02.202067 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 64, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:02.208364 24787 storage/replica_command.go:284 [n3,split,s3,r52/1:/{Table/64-Max}] initiating a split of this range at key /Table/64/1 [r53] (zone config) I190903 18:32:02.223381 24299 storage/replica_command.go:284 [n3,split,s3,r53/1:/{Table/64/1-Max}] initiating a split of this range at key /Table/64/2 [r54] (zone config) I190903 18:32:02.252762 24551 storage/store.go:2593 [n3,replicaGC,s3,r46/1:/Table/{58/2-60}] removing replica r46/1 I190903 18:32:02.253223 24551 storage/replica_destroy.go:152 [n3,replicaGC,s3,r46/1:/Table/{58/2-60}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:02.266782 25028 storage/replica_command.go:1430 [n3,replicate,s3,r53/1:/Table/64/{1-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r53:/Table/64/{1-2} [(n3,s3):1, next=2, gen=31] I190903 18:32:02.277725 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:02.284475 25221 storage/replica_command.go:284 [n3,split,s3,r53/1:/Table/64/{1-2}] initiating a split of this range at key /Table/64/1/3 [r55] (zone config) I190903 18:32:02.296412 25028 storage/replica_raft.go:291 [n3,s3,r53/1:/Table/64/{1-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:02.298426 25028 storage/store_snapshot.go:995 [n3,replicate,s3,r53/1:/Table/64/{1-2}] sending LEARNER snapshot 3f8f5b45 at applied index 18 I190903 18:32:02.298605 25028 storage/store_snapshot.go:1038 [n3,replicate,s3,r53/1:/Table/64/{1-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:02.299875 25226 storage/replica_raftstorage.go:793 [n1,s1,r53/2:{-}] applying LEARNER snapshot [id=3f8f5b45 index=18] I190903 18:32:02.302171 25226 storage/replica_raftstorage.go:814 [n1,s1,r53/2:/Table/64/{1-2}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=3f8f5b45 index=18] I190903 18:32:02.302587 25028 storage/replica_command.go:1430 [n3,replicate,s3,r53/1:/Table/64/{1-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r53:/Table/64/{1-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=32] I190903 18:32:02.303548 25221 storage/split_queue.go:149 [n3,split,s3,r53/1:/Table/64/{1-2}] split saw concurrent descriptor modification; maybe retrying I190903 18:32:02.303698 25010 storage/replica_command.go:284 [n3,split,s3,r53/1:/Table/64/{1-2}] initiating a split of this range at key /Table/64/1/3 [r56] (zone config) I190903 18:32:02.305372 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:02.305782 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.330234 25028 storage/replica_raft.go:291 [n3,s3,r53/1:/Table/64/{1-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:02.336186 25028 storage/replica_command.go:1430 [n3,replicate,s3,r53/1:/Table/64/{1-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r53:/Table/64/{1-2} [(n3,s3):1, (n1,s1):2, next=3, gen=33] I190903 18:32:02.352871 25010 storage/split_queue.go:149 [n3,split,s3,r53/1:/Table/64/{1-2}] split saw concurrent descriptor modification; maybe retrying I190903 18:32:02.353055 25120 storage/replica_command.go:284 [n3,split,s3,r53/1:/Table/64/{1-2}] initiating a split of this range at key /Table/64/1/3 [r57] (zone config) I190903 18:32:02.378687 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.385230 25028 storage/replica_raft.go:291 [n3,s3,r53/1:/Table/64/{1-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:02.385319 25028 storage/replica_raft.go:304 [n3,s3,r53/1:/Table/64/{1-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:02.397815 25028 storage/replica_command.go:979 [n3,replicate,s3,r53/1:/Table/64/{1-2}] could not promote [] to voter, rolling back: change replicas of r53 failed: storage/replica_raft.go:305: [n3,s3,r53/1:/Table/64/{1-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:02.397878 25028 storage/queue.go:1027 [n3,replicate,s3,r53/1:/Table/64/{1-2}] change replicas of r53 failed: storage/replica_raft.go:305: [n3,s3,r53/1:/Table/64/{1-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:02.409400 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.418006 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.438420 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:02.449994 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.471981 25498 storage/replica_command.go:284 [n3,split,s3,r57/1:/Table/64/{1/3-2}] initiating a split of this range at key /Table/64/1/4 [r58] (zone config) I190903 18:32:02.478939 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:02.485610 25959 storage/queue.go:1027 [n1,replicate,s1,r53/2:/Table/64/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:28] I190903 18:32:02.506810 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.511535 26215 storage/replica_command.go:284 [n3,split,s3,r58/1:/Table/64/{1/4-2}] initiating a split of this range at key /Table/64/1/5 [r59] (zone config) I190903 18:32:02.515664 25941 storage/replica_command.go:1430 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r57:/Table/64/1/{3-4} [(n3,s3):1, (n1,s1):2, next=3, gen=35] I190903 18:32:02.532025 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.537281 25941 storage/replica_raft.go:291 [n3,s3,r57/1:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 18:32:02.538566 25941 storage/replica_command.go:1430 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r57:/Table/64/1/{3-4} [(n3,s3):1, next=3, gen=36] I190903 18:32:02.545108 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:02.549627 25941 storage/replica_raft.go:291 [n3,s3,r57/1:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):1 (n2,s2):3LEARNER] next=4 I190903 18:32:02.549666 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.551262 25941 storage/store_snapshot.go:995 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] sending LEARNER snapshot 1d30e24a at applied index 20 I190903 18:32:02.551626 25941 storage/store_snapshot.go:1038 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:02.552962 26224 storage/replica_raftstorage.go:793 [n2,s2,r57/3:{-}] applying LEARNER snapshot [id=1d30e24a index=20] I190903 18:32:02.558233 26135 storage/store.go:2593 [n1,replicaGC,s1,r57/2:/Table/64/1/{3-4}] removing replica r57/2 I190903 18:32:02.560381 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.560555 26135 storage/replica_destroy.go:152 [n1,replicaGC,s1,r57/2:/Table/64/1/{3-4}] removed 6 (0+6) keys in 2ms [clear=2ms commit=0ms] I190903 18:32:02.563358 26224 storage/replica_raftstorage.go:814 [n2,s2,r57/3:/Table/64/1/{3-4}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=1d30e24a index=20] I190903 18:32:02.563818 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.568145 25941 storage/replica_command.go:1430 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] change replicas (add [(n2,s2):3] remove []): existing descriptor r57:/Table/64/1/{3-4} [(n3,s3):1, (n2,s2):3LEARNER, next=4, gen=37] I190903 18:32:02.583680 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.587212 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.590404 25941 storage/replica_raft.go:291 [n3,s3,r57/1:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):1 (n2,s2):3] next=4 I190903 18:32:02.591306 25941 storage/replica_command.go:1430 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r57:/Table/64/1/{3-4} [(n3,s3):1, (n2,s2):3, next=4, gen=38] I190903 18:32:02.613219 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.617389 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.627541 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.631536 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.657129 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:02.663660 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.663828 25941 storage/replica_raft.go:291 [n3,s3,r57/1:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 E190903 18:32:02.664121 25941 storage/replica_raft.go:304 [n3,s3,r57/1:/Table/64/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:02.667238 26746 storage/replica_command.go:1430 [n1,replicate,s1,r53/2:/Table/64/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r53:/Table/64/1{-/3} [(n3,s3):1, (n1,s1):2, next=3, gen=34] I190903 18:32:02.668521 25941 storage/replica_command.go:979 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r57 failed: storage/replica_raft.go:305: [n3,s3,r57/1:/Table/64/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:02.668591 25941 storage/queue.go:1027 [n3,replicate,s3,r57/1:/Table/64/1/{3-4}] change replicas of r57 failed: storage/replica_raft.go:305: [n3,s3,r57/1:/Table/64/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:02.669009 26706 storage/replica_command.go:1430 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r58:/Table/64/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=36] I190903 18:32:02.675419 26746 storage/replica_raft.go:291 [n1,s1,r53/2:/Table/64/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:02.678379 26706 storage/replica_raft.go:291 [n3,s3,r58/1:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 18:32:02.679971 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.681317 26839 storage/replica_command.go:1430 [n3,replicate,s3,r47/1:/Table/60{-/1}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r47:/Table/60{-/1} [(n3,s3):1, next=2, gen=17] I190903 18:32:02.685878 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.688832 26750 storage/store.go:2593 [n1,replicaGC,s1,r58/2:/Table/64/1/{4-5}] removing replica r58/2 I190903 18:32:02.688881 26868 storage/store.go:2593 [n3,replicaGC,s3,r53/1:/Table/64/1{-/3}] removing replica r53/1 I190903 18:32:02.689179 26750 storage/replica_destroy.go:152 [n1,replicaGC,s1,r58/2:/Table/64/1/{4-5}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:02.689309 26868 storage/replica_destroy.go:152 [n3,replicaGC,s3,r53/1:/Table/64/1{-/3}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:02.698370 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.700546 26839 storage/replica_raft.go:291 [n3,s3,r47/1:/Table/60{-/1}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:02.700860 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.701611 26839 storage/store_snapshot.go:995 [n3,replicate,s3,r47/1:/Table/60{-/1}] sending LEARNER snapshot bc1d5060 at applied index 24 I190903 18:32:02.701786 26839 storage/store_snapshot.go:1038 [n3,replicate,s3,r47/1:/Table/60{-/1}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:02.708069 26980 storage/replica_raftstorage.go:793 [n2,s2,r47/2:{-}] applying LEARNER snapshot [id=bc1d5060 index=24] I190903 18:32:02.709866 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:02.715267 26980 storage/replica_raftstorage.go:814 [n2,s2,r47/2:/Table/60{-/1}] applied LEARNER snapshot [total=7ms ingestion=4@1ms id=bc1d5060 index=24] I190903 18:32:02.718811 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.721417 26839 storage/replica_command.go:1430 [n3,replicate,s3,r47/1:/Table/60{-/1}] change replicas (add [(n2,s2):2] remove []): existing descriptor r47:/Table/60{-/1} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=18] I190903 18:32:02.750875 26839 storage/replica_raft.go:291 [n3,s3,r47/1:/Table/60{-/1}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:02.751460 26839 storage/replica_command.go:1430 [n3,replicate,s3,r47/1:/Table/60{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r47:/Table/60{-/1} [(n3,s3):1, (n2,s2):2, next=3, gen=19] I190903 18:32:02.765212 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.780306 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.782308 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.785722 26839 storage/replica_raft.go:291 [n3,s3,r47/1:/Table/60{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:02.785786 26839 storage/replica_raft.go:304 [n3,s3,r47/1:/Table/60{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:02.786126 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.789468 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.789752 26839 storage/replica_command.go:979 [n3,replicate,s3,r47/1:/Table/60{-/1}] could not promote [] to voter, rolling back: change replicas of r47 failed: storage/replica_raft.go:305: [n3,s3,r47/1:/Table/60{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:02.789802 26839 storage/queue.go:1027 [n3,replicate,s3,r47/1:/Table/60{-/1}] change replicas of r47 failed: storage/replica_raft.go:305: [n3,s3,r47/1:/Table/60{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:02.793153 27112 storage/queue.go:1027 [n1,replicate,s1,r59/2:/Table/64/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:13] I190903 18:32:02.799396 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 E190903 18:32:02.807346 27234 storage/queue.go:1027 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 3*:29] I190903 18:32:02.840134 27429 storage/replica_command.go:1430 [n1,replicate,s1,r59/2:/Table/64/{1/5-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r59:/Table/64/{1/5-2} [(n3,s3):1, (n1,s1):2, next=3, gen=36] I190903 18:32:02.846181 27429 storage/replica_raft.go:291 [n1,s1,r59/2:/Table/64/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:02.874316 27374 storage/store.go:2593 [n3,replicaGC,s3,r59/1:/Table/64/{1/5-2}] removing replica r59/1 I190903 18:32:02.876543 27523 storage/replica_command.go:1430 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r57:/Table/64/1/{3-4} [(n3,s3):1, (n2,s2):3, next=4, gen=38] I190903 18:32:02.879358 27374 storage/replica_destroy.go:152 [n3,replicaGC,s3,r59/1:/Table/64/{1/5-2}] removed 5 (0+5) keys in 1ms [clear=0ms commit=1ms] I190903 18:32:02.879442 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:02.890934 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:02.896598 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:02.899095 27523 storage/replica_raft.go:291 [n2,s2,r57/3:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 I190903 18:32:02.911789 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:02.925464 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=64 (single col list partitioning - DEFAULT) version=2 mtime=2019-09-03 18:32:02.924118156 +0000 UTC I190903 18:32:02.936910 27684 storage/store.go:2593 [n3,replicaGC,s3,r57/1:/Table/64/1/{3-4}] removing replica r57/1 I190903 18:32:02.953321 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 1}] I190903 18:32:02.955615 27684 storage/replica_destroy.go:152 [n3,replicaGC,s3,r57/1:/Table/64/1/{3-4}] removed 11 (0+11) keys in 9ms [clear=9ms commit=0ms] I190903 18:32:02.957167 27749 sql/lease.go:923 new lease: 64("single col list partitioning - DEFAULT") ver=2:1567535830.440200905,0, refcount=0 I190903 18:32:03.013847 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=64 (single col list partitioning - DEFAULT) version=3 mtime=2019-09-03 18:32:03.013672758 +0000 UTC I190903 18:32:03.020730 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 2}] I190903 18:32:03.045808 28090 sql/lease.go:923 new lease: 64("single col list partitioning - DEFAULT") ver=3:1567535830.441200905,0, refcount=0 E190903 18:32:03.063278 28360 storage/queue.go:1027 [n2,replicate,s2,r47/2:/Table/60{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:31] I190903 18:32:03.120459 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=64 (single col list partitioning - DEFAULT) version=4 mtime=2019-09-03 18:32:03.117837866 +0000 UTC I190903 18:32:03.139647 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 64, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:ALTER TABLE "single col list partitioning - DEFAULT" PARTITION BY LIST (a) (PARTITION p4 VALUES IN ((4)), PARTITION p5 VALUES IN ((5)), PARTITION pd VALUES IN ((DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:03.141979 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 3}] I190903 18:32:03.144010 28525 sql/lease.go:923 new lease: 64("single col list partitioning - DEFAULT") ver=4:1567535830.442200905,0, refcount=0 I190903 18:32:03.147624 27310 storage/replica_command.go:1430 [n2,s2,r57/3:/Table/64/1/{3-4}] change replicas (add [(n1,s1):4LEARNER] remove []): existing descriptor r57:/Table/64/1/{3-4} [(n2,s2):3, next=4, gen=39] I190903 18:32:03.180261 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:03.181614 27310 storage/replica_raft.go:291 [n2,s2,r57/3:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n1,s1):4LEARNER]: after=[(n2,s2):3 (n1,s1):4LEARNER] next=5 I190903 18:32:03.184518 27310 storage/store_snapshot.go:995 [n2,s2,r57/3:/Table/64/1/{3-4}] sending LEARNER snapshot 0da1b273 at applied index 35 I190903 18:32:03.184710 27310 storage/store_snapshot.go:1038 [n2,s2,r57/3:/Table/64/1/{3-4}] streamed snapshot to (n1,s1):4LEARNER: kv pairs: 19, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:03.185991 28880 storage/replica_raftstorage.go:793 [n1,s1,r57/4:{-}] applying LEARNER snapshot [id=0da1b273 index=35] I190903 18:32:03.186609 28727 storage/replica_command.go:1430 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] change replicas (add [] remove [(n1,s1):4LEARNER]): existing descriptor r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):4LEARNER, next=5, gen=40] I190903 18:32:03.195048 28107 storage/replica_command.go:1430 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r58:/Table/64/1/{4-5} [(n3,s3):1, next=3, gen=37] I190903 18:32:03.199219 28880 storage/replica_raftstorage.go:814 [n1,s1,r57/4:/Table/64/1/{3-4}] applied LEARNER snapshot [total=13ms ingestion=4@13ms id=0da1b273 index=35] I190903 18:32:03.202411 28727 storage/replica_raft.go:291 [n2,s2,r57/3:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):4LEARNER]: after=[(n2,s2):3] next=5 I190903 18:32:03.203837 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:03.211014 29336 storage/replica_command.go:284 [n1,split,s1,r59/2:/Table/64/{1/5-2}] initiating a split of this range at key /Table/64/1/6 [r33] (zone config) I190903 18:32:03.223400 27454 storage/replica_command.go:1430 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r59:/Table/64/1/{5-6} [(n1,s1):2, next=3, gen=38] I190903 18:32:03.224696 28107 storage/replica_raft.go:291 [n3,s3,r58/1:/Table/64/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):1 (n2,s2):3LEARNER] next=4 I190903 18:32:03.227158 28107 storage/store_snapshot.go:995 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] sending LEARNER snapshot 3a7b1428 at applied index 22 I190903 18:32:03.227340 28107 storage/store_snapshot.go:1038 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:03.228442 29406 storage/replica_raftstorage.go:793 [n2,s2,r58/3:{-}] applying LEARNER snapshot [id=3a7b1428 index=22] I190903 18:32:03.229787 29406 storage/replica_raftstorage.go:814 [n2,s2,r58/3:/Table/64/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=3a7b1428 index=22] I190903 18:32:03.230221 28107 storage/replica_command.go:1430 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r58:/Table/64/1/{4-5} [(n3,s3):1, (n2,s2):3LEARNER, next=4, gen=38] I190903 18:32:03.238202 27454 storage/replica_raft.go:291 [n1,s1,r59/2:/Table/64/1/{5-6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:03.245111 27454 storage/store_snapshot.go:995 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] sending LEARNER snapshot aea87161 at applied index 21 I190903 18:32:03.245442 27454 storage/store_snapshot.go:1038 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:03.247045 29392 storage/replica_raftstorage.go:793 [n3,s3,r59/3:{-}] applying LEARNER snapshot [id=aea87161 index=21] I190903 18:32:03.247938 28727 storage/replica_command.go:1430 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] change replicas (add [(n1,s1):5LEARNER] remove []): existing descriptor r57:/Table/64/1/{3-4} [(n2,s2):3, next=5, gen=41] I190903 18:32:03.248170 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:03.249885 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.253438 29031 storage/store.go:2593 [n1,replicaGC,s1,r57/4:/Table/64/1/{3-4}] removing replica r57/4 I190903 18:32:03.253573 28107 storage/replica_raft.go:291 [n3,s3,r58/1:/Table/64/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):1 (n2,s2):3] next=4 I190903 18:32:03.253723 29031 storage/replica_destroy.go:152 [n1,replicaGC,s1,r57/4:/Table/64/1/{3-4}] removed 13 (0+13) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:03.254756 27310 storage/replica_command.go:979 [n2,s2,r57/3:/Table/64/1/{3-4}] could not promote [n1,s1] to voter, rolling back: change replicas of r57 failed: descriptor changed: [expected] r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):4LEARNER, next=5, gen=40] != [actual] r57:/Table/64/1/{3-4} [(n2,s2):3, next=5, gen=41] E190903 18:32:03.255862 28700 storage/queue.go:1027 [n1,merge,s1,r53/2:/Table/64/1{-/3}] while adding target n1,s1: change replicas of r57 failed: descriptor changed: [expected] r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):4LEARNER, next=5, gen=40] != [actual] r57:/Table/64/1/{3-4} [(n2,s2):3, next=5, gen=41] I190903 18:32:03.262321 28107 storage/replica_command.go:1430 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r58:/Table/64/1/{4-5} [(n3,s3):1, (n2,s2):3, next=4, gen=39] I190903 18:32:03.275875 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.299840 29392 storage/replica_raftstorage.go:814 [n3,s3,r59/3:/Table/64/1/{5-6}] applied LEARNER snapshot [total=52ms ingestion=4@52ms id=aea87161 index=21] I190903 18:32:03.305279 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:03.307997 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.311360 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:03.311651 28727 storage/replica_raft.go:291 [n2,s2,r57/3:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n1,s1):5LEARNER]: after=[(n2,s2):3 (n1,s1):5LEARNER] next=6 I190903 18:32:03.312872 28727 storage/store_snapshot.go:995 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] sending LEARNER snapshot b4891ee7 at applied index 40 I190903 18:32:03.313077 28727 storage/store_snapshot.go:1038 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] streamed snapshot to (n1,s1):5LEARNER: kv pairs: 23, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:03.314226 29837 storage/replica_raftstorage.go:793 [n1,s1,r57/5:{-}] applying LEARNER snapshot [id=b4891ee7 index=40] I190903 18:32:03.316122 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.317584 29837 storage/replica_raftstorage.go:814 [n1,s1,r57/5:/Table/64/1/{3-4}] applied LEARNER snapshot [total=3ms ingestion=4@3ms id=b4891ee7 index=40] I190903 18:32:03.319897 27454 storage/replica_command.go:1430 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r59:/Table/64/1/{5-6} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=39] I190903 18:32:03.322957 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.326287 28107 storage/replica_raft.go:291 [n3,s3,r58/1:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 E190903 18:32:03.326351 28107 storage/replica_raft.go:304 [n3,s3,r58/1:/Table/64/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:03.327678 28727 storage/replica_command.go:1430 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] change replicas (add [(n1,s1):5] remove []): existing descriptor r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):5LEARNER, next=6, gen=42] E190903 18:32:03.329280 29246 storage/queue.go:1027 [n1,merge,s1,r53/2:/Table/64/1{-/3}] while adding target n1,s1: change replicas of r57 failed: descriptor changed: [expected] r57:/Table/64/1/{3-4} [(n2,s2):3, next=5, gen=41] != [actual] r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):5LEARNER, next=6, gen=42] I190903 18:32:03.329596 28107 storage/replica_command.go:979 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r58 failed: storage/replica_raft.go:305: [n3,s3,r58/1:/Table/64/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:03.331354 28107 storage/queue.go:1027 [n3,replicate,s3,r58/1:/Table/64/1/{4-5}] change replicas of r58 failed: storage/replica_raft.go:305: [n3,s3,r58/1:/Table/64/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:03.332258 27454 storage/replica_raft.go:291 [n1,s1,r59/2:/Table/64/1/{5-6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 18:32:03.372446 27454 storage/replica_command.go:1430 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r59:/Table/64/1/{5-6} [(n1,s1):2, (n3,s3):3, next=4, gen=40] I190903 18:32:03.376366 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.382427 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.382554 28727 storage/replica_raft.go:291 [n2,s2,r57/3:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n1,s1):5]: after=[(n2,s2):3 (n1,s1):5] next=6 I190903 18:32:03.383122 28727 storage/replica_command.go:1430 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] change replicas (add [] remove [(n2,s2):3]): existing descriptor r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):5, next=6, gen=43] I190903 18:32:03.384374 27454 storage/replica_raft.go:291 [n1,s1,r59/2:/Table/64/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 18:32:03.384437 27454 storage/replica_raft.go:304 [n1,s1,r59/2:/Table/64/1/{5-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:03.394785 27454 storage/replica_command.go:979 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] could not promote [] to voter, rolling back: change replicas of r59 failed: storage/replica_raft.go:305: [n1,s1,r59/2:/Table/64/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:03.394824 27454 storage/queue.go:1027 [n1,replicate,s1,r59/2:/Table/64/1/{5-6}] change replicas of r59 failed: storage/replica_raft.go:305: [n1,s1,r59/2:/Table/64/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:03.398838 28727 storage/replica_raft.go:291 [n2,s2,r57/3:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):3]: after=[(n1,s1):5] next=6 E190903 18:32:03.398943 28727 storage/replica_raft.go:304 [n2,s2,r57/3:/Table/64/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n1,s1):5] next=6 to remove self (leaseholder) I190903 18:32:03.400257 28727 storage/replica_command.go:979 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r57 failed: storage/replica_raft.go:305: [n2,s2,r57/3:/Table/64/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n1,s1):5] next=6 to remove self (leaseholder) E190903 18:32:03.400293 28727 storage/queue.go:1027 [n2,replicate,s2,r57/3:/Table/64/1/{3-4}] change replicas of r57 failed: storage/replica_raft.go:305: [n2,s2,r57/3:/Table/64/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n1,s1):5] next=6 to remove self (leaseholder) I190903 18:32:03.402437 30243 storage/replica_command.go:1430 [n2,replicate,s2,r58/3:/Table/64/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r58:/Table/64/1/{4-5} [(n3,s3):1, (n2,s2):3, next=4, gen=39] I190903 18:32:03.428597 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:03.430358 30243 storage/replica_raft.go:291 [n2,s2,r58/3:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):3] next=4 I190903 18:32:03.434332 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.464404 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.469456 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:03.473252 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.489860 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.490636 30319 storage/replica_command.go:1430 [n2,replicate,s2,r47/2:/Table/60{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r47:/Table/60{-/1} [(n3,s3):1, (n2,s2):2, next=3, gen=19] I190903 18:32:03.517508 30322 storage/store.go:2593 [n3,replicaGC,s3,r58/1:/Table/64/1/{4-5}] removing replica r58/1 I190903 18:32:03.517872 30322 storage/replica_destroy.go:152 [n3,replicaGC,s3,r58/1:/Table/64/1/{4-5}] removed 11 (0+11) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:03.518142 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:03.526712 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 E190903 18:32:03.551205 30145 storage/queue.go:1027 [n1,replicate,s1,r57/5:/Table/64/1/{3-4}] no removable replicas from range that needs a removal: [3:0, 5*:47] I190903 18:32:03.551705 30319 storage/replica_raft.go:291 [n2,s2,r47/2:/Table/60{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:03.578587 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.585002 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:03.593097 30862 storage/replica_command.go:1430 [n1,s1,r57/5:/Table/64/1/{3-4}] change replicas (add [] remove [(n2,s2):3]): existing descriptor r57:/Table/64/1/{3-4} [(n2,s2):3, (n1,s1):5, next=6, gen=43] I190903 18:32:03.598767 30862 storage/replica_raft.go:291 [n1,s1,r57/5:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):3]: after=[(n1,s1):5] next=6 I190903 18:32:03.599158 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:03.602555 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.602825 30648 storage/replica_command.go:609 [n1,merge,s1,r53/2:/Table/64/1{-/3}] initiating a merge of r57:/Table/64/1/{3-4} [(n1,s1):5, next=6, gen=44] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:03.605146 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.609000 30685 storage/store.go:2593 [n2,replicaGC,s2,r57/3:/Table/64/1/{3-4}] removing replica r57/3 I190903 18:32:03.609564 30685 storage/replica_destroy.go:152 [n2,replicaGC,s2,r57/3:/Table/64/1/{3-4}] removed 12 (0+12) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:03.611188 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:03.617346 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:03.621136 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.622845 200 storage/store.go:2593 [n1,s1,r53/2:/Table/64/1{-/3}] removing replica r57/5 I190903 18:32:03.630641 30764 storage/store.go:2593 [n3,replicaGC,s3,r47/1:/Table/60{-/1}] removing replica r47/1 I190903 18:32:03.630987 30764 storage/replica_destroy.go:152 [n3,replicaGC,s3,r47/1:/Table/60{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:03.685542 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:03.692623 23523 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 E190903 18:32:03.699734 30917 storage/queue.go:1027 [n3,replicate,s3,r59/3:/Table/64/1/{5-6}] no removable replicas from range that needs a removal: [2:0, 3*:28] === RUN TestRepartitioning/single_col_list_partitioning_-_DEFAULT/single_col_list_partitioning I190903 18:32:03.709106 31245 storage/replica_command.go:1430 [n3,replicate,s3,r59/3:/Table/64/1/{5-6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r59:/Table/64/1/{5-6} [(n1,s1):2, (n3,s3):3, next=4, gen=40] I190903 18:32:03.717247 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=64 (single col list partitioning - DEFAULT) version=5 mtime=2019-09-03 18:32:03.706667159 +0000 UTC I190903 18:32:03.749941 31245 storage/replica_raft.go:291 [n3,s3,r59/3:/Table/64/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:03.751495 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 63, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning - DEFAULT"]} I190903 18:32:03.753718 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 4}] I190903 18:32:03.768456 31604 storage/store.go:2593 [n1,replicaGC,s1,r59/2:/Table/64/1/{5-6}] removing replica r59/2 I190903 18:32:03.768921 31604 storage/replica_destroy.go:152 [n1,replicaGC,s1,r59/2:/Table/64/1/{5-6}] removed 11 (0+11) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:03.802041 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=64 (single col list partitioning - DEFAULT) version=6 mtime=2019-09-03 18:32:03.801824903 +0000 UTC I190903 18:32:03.840577 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 65, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:03.847368 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 66, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:CREATE TABLE "single col list partitioning - DEFAULT" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p4 VALUES IN (4), PARTITION p5 VALUES IN (5), PARTITION pd VALUES IN (DEFAULT)) User:root} I190903 18:32:03.851894 32301 storage/replica_command.go:284 [n3,split,s3,r54/1:/{Table/64/2-Max}] initiating a split of this range at key /Table/66 [r60] (zone config) I190903 18:32:03.873896 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:03.880015 32623 storage/replica_command.go:284 [n3,split,s3,r60/1:/{Table/66-Max}] initiating a split of this range at key /Table/66/1/4 [r61] (zone config) I190903 18:32:03.882695 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:03.914951 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:03.915357 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.921272 33139 storage/replica_command.go:284 [n3,split,s3,r61/1:/{Table/66/1/4-Max}] initiating a split of this range at key /Table/66/1/5 [r62] (zone config) I190903 18:32:03.921577 33127 storage/replica_command.go:284 [n3,split,s3,r60/1:/Table/66{-/1/4}] initiating a split of this range at key /Table/66/1 [r63] (zone config) I190903 18:32:03.948072 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:03.956297 33269 storage/replica_command.go:1430 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r61:/Table/66/1/{4-5} [(n3,s3):1, next=2, gen=34] I190903 18:32:03.957351 33295 storage/replica_command.go:284 [n3,split,s3,r62/1:/{Table/66/1/5-Max}] initiating a split of this range at key /Table/66/1/6 [r64] (zone config) I190903 18:32:03.970491 33269 storage/replica_raft.go:291 [n3,s3,r61/1:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:03.999644 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:04.003251 33269 storage/store_snapshot.go:995 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] sending LEARNER snapshot 292ddf06 at applied index 17 I190903 18:32:04.003468 33269 storage/store_snapshot.go:1038 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:04.004873 33361 storage/replica_raftstorage.go:793 [n2,s2,r61/2:{-}] applying LEARNER snapshot [id=292ddf06 index=17] I190903 18:32:04.006489 33361 storage/replica_raftstorage.go:814 [n2,s2,r61/2:/Table/66/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=292ddf06 index=17] I190903 18:32:04.010282 33269 storage/replica_command.go:1430 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r61:/Table/66/1/{4-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=35] I190903 18:32:04.049779 33273 storage/replica_command.go:284 [n3,split,s3,r64/1:/{Table/66/1/6-Max}] initiating a split of this range at key /Table/66/2 [r65] (zone config) I190903 18:32:04.050898 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.067615 33269 storage/replica_raft.go:291 [n3,s3,r61/1:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:04.068247 33269 storage/replica_command.go:1430 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r61:/Table/66/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=36] I190903 18:32:04.091774 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.103595 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.103982 33269 storage/replica_raft.go:291 [n3,s3,r61/1:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:04.104100 33269 storage/replica_raft.go:304 [n3,s3,r61/1:/Table/66/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:04.105573 33269 storage/replica_command.go:979 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r61 failed: storage/replica_raft.go:305: [n3,s3,r61/1:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:04.105617 33269 storage/queue.go:1027 [n3,replicate,s3,r61/1:/Table/66/1/{4-5}] change replicas of r61 failed: storage/replica_raft.go:305: [n3,s3,r61/1:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:04.106050 33732 storage/replica_command.go:1430 [n3,replicate,s3,r60/1:/Table/66{-/1}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r60:/Table/66{-/1} [(n3,s3):1, next=2, gen=34] I190903 18:32:04.108505 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.117607 33732 storage/replica_raft.go:291 [n3,s3,r60/1:/Table/66{-/1}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:04.118777 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.119434 33732 storage/store_snapshot.go:995 [n3,replicate,s3,r60/1:/Table/66{-/1}] sending LEARNER snapshot a1d6b2ae at applied index 22 I190903 18:32:04.119591 33732 storage/store_snapshot.go:1038 [n3,replicate,s3,r60/1:/Table/66{-/1}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:04.120717 33740 storage/replica_raftstorage.go:793 [n1,s1,r60/2:{-}] applying LEARNER snapshot [id=a1d6b2ae index=22] I190903 18:32:04.127040 33740 storage/replica_raftstorage.go:814 [n1,s1,r60/2:/Table/66{-/1}] applied LEARNER snapshot [total=6ms ingestion=4@6ms id=a1d6b2ae index=22] I190903 18:32:04.129970 33732 storage/replica_command.go:1430 [n3,replicate,s3,r60/1:/Table/66{-/1}] change replicas (add [(n1,s1):2] remove []): existing descriptor r60:/Table/66{-/1} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=35] I190903 18:32:04.140667 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.142465 33732 storage/replica_raft.go:291 [n3,s3,r60/1:/Table/66{-/1}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:04.143556 33732 storage/replica_command.go:1430 [n3,replicate,s3,r60/1:/Table/66{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r60:/Table/66{-/1} [(n3,s3):1, (n1,s1):2, next=3, gen=36] I190903 18:32:04.151758 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.157728 33732 storage/replica_raft.go:291 [n3,s3,r60/1:/Table/66{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:04.157774 33732 storage/replica_raft.go:304 [n3,s3,r60/1:/Table/66{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:04.160609 33732 storage/replica_command.go:979 [n3,replicate,s3,r60/1:/Table/66{-/1}] could not promote [] to voter, rolling back: change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/1:/Table/66{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:04.160670 33732 storage/queue.go:1027 [n3,replicate,s3,r60/1:/Table/66{-/1}] change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/1:/Table/66{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:04.161182 33868 storage/replica_command.go:1430 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r63:/Table/66/1{-/4} [(n3,s3):1, next=2, gen=34] I190903 18:32:04.162373 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.167164 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.174871 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.176696 33868 storage/replica_raft.go:291 [n3,s3,r63/1:/Table/66/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:04.189848 33868 storage/store_snapshot.go:995 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] sending LEARNER snapshot ac413a6a at applied index 15 I190903 18:32:04.190025 33868 storage/store_snapshot.go:1038 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:04.191280 33908 storage/replica_raftstorage.go:793 [n1,s1,r63/2:{-}] applying LEARNER snapshot [id=ac413a6a index=15] I190903 18:32:04.192618 33908 storage/replica_raftstorage.go:814 [n1,s1,r63/2:/Table/66/1{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ac413a6a index=15] I190903 18:32:04.193165 33868 storage/replica_command.go:1430 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r63:/Table/66/1{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=35] I190903 18:32:04.194148 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.207539 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.212163 158 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:04.216631 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.216843 33868 storage/replica_raft.go:291 [n3,s3,r63/1:/Table/66/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:04.217804 33868 storage/replica_command.go:1430 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r63:/Table/66/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=36] I190903 18:32:04.226590 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:04.229742 33868 storage/replica_raft.go:291 [n3,s3,r63/1:/Table/66/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:04.229782 33868 storage/replica_raft.go:304 [n3,s3,r63/1:/Table/66/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:04.231083 33868 storage/replica_command.go:979 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] could not promote [] to voter, rolling back: change replicas of r63 failed: storage/replica_raft.go:305: [n3,s3,r63/1:/Table/66/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:04.231124 33868 storage/queue.go:1027 [n3,replicate,s3,r63/1:/Table/66/1{-/4}] change replicas of r63 failed: storage/replica_raft.go:305: [n3,s3,r63/1:/Table/66/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:04.235414 34275 storage/replica_command.go:1430 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r64:/Table/66/{1/6-2} [(n3,s3):1, next=2, gen=36] E190903 18:32:04.236585 34310 storage/queue.go:1027 [n1,replicate,s1,r60/2:/Table/66{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 18:32:04.238377 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.243248 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:04.249629 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.252632 34275 storage/replica_raft.go:291 [n3,s3,r64/1:/Table/66/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:04.254817 34275 storage/store_snapshot.go:995 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] sending LEARNER snapshot 7b717ee5 at applied index 18 I190903 18:32:04.255552 34275 storage/store_snapshot.go:1038 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:04.256752 34387 storage/replica_raftstorage.go:793 [n1,s1,r64/2:{-}] applying LEARNER snapshot [id=7b717ee5 index=18] I190903 18:32:04.259679 34387 storage/replica_raftstorage.go:814 [n1,s1,r64/2:/Table/66/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=7b717ee5 index=18] I190903 18:32:04.257275 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.261663 34275 storage/replica_command.go:1430 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r64:/Table/66/{1/6-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=37] I190903 18:32:04.274166 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:04.278302 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:04.278559 34275 storage/replica_raft.go:291 [n3,s3,r64/1:/Table/66/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:04.280750 34275 storage/replica_command.go:1430 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r64:/Table/66/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=38] I190903 18:32:04.287490 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.296620 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.299953 34275 storage/replica_raft.go:291 [n3,s3,r64/1:/Table/66/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:04.300004 34275 storage/replica_raft.go:304 [n3,s3,r64/1:/Table/66/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:04.301287 34275 storage/replica_command.go:979 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r64 failed: storage/replica_raft.go:305: [n3,s3,r64/1:/Table/66/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:04.301332 34275 storage/queue.go:1027 [n3,replicate,s3,r64/1:/Table/66/{1/6-2}] change replicas of r64 failed: storage/replica_raft.go:305: [n3,s3,r64/1:/Table/66/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:04.301853 34474 storage/replica_command.go:1430 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r65:/{Table/66/2-Max} [(n3,s3):1, next=2, gen=36] I190903 18:32:04.304088 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:04.310769 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.310981 34474 storage/replica_raft.go:291 [n3,s3,r65/1:/{Table/66/2-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:04.314046 34474 storage/store_snapshot.go:995 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] sending LEARNER snapshot b55ec2bb at applied index 15 I190903 18:32:04.314202 34474 storage/store_snapshot.go:1038 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:04.315353 34523 storage/replica_raftstorage.go:793 [n2,s2,r65/2:{-}] applying LEARNER snapshot [id=b55ec2bb index=15] I190903 18:32:04.316777 34523 storage/replica_raftstorage.go:814 [n2,s2,r65/2:/{Table/66/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=b55ec2bb index=15] I190903 18:32:04.317264 34474 storage/replica_command.go:1430 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r65:/{Table/66/2-Max} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=37] I190903 18:32:04.319686 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.327575 34474 storage/replica_raft.go:291 [n3,s3,r65/1:/{Table/66/2-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:04.328573 34474 storage/replica_command.go:1430 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r65:/{Table/66/2-Max} [(n3,s3):1, (n2,s2):2, next=3, gen=38] I190903 18:32:04.337055 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.344293 34474 storage/replica_raft.go:291 [n3,s3,r65/1:/{Table/66/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:04.344360 34474 storage/replica_raft.go:304 [n3,s3,r65/1:/{Table/66/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:04.345643 34474 storage/replica_command.go:979 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] could not promote [] to voter, rolling back: change replicas of r65 failed: storage/replica_raft.go:305: [n3,s3,r65/1:/{Table/66/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:04.345719 34474 storage/queue.go:1027 [n3,replicate,s3,r65/1:/{Table/66/2-Max}] change replicas of r65 failed: storage/replica_raft.go:305: [n3,s3,r65/1:/{Table/66/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:04.357289 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.396954 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 E190903 18:32:04.407303 35027 storage/queue.go:1027 [n2,replicate,s2,r65/2:/{Table/66/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:21] E190903 18:32:04.437179 35107 storage/queue.go:1027 [n1,replicate,s1,r64/2:/Table/66/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:04.456848 35147 storage/replica_command.go:1430 [n1,replicate,s1,r64/2:/Table/66/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r64:/Table/66/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=38] I190903 18:32:04.461807 35147 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/66/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:04.474953 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 E190903 18:32:04.475755 35106 storage/queue.go:1027 [n1,replicate,s1,r63/2:/Table/66/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:04.486554 35203 storage/store.go:2593 [n3,replicaGC,s3,r64/1:/Table/66/{1/6-2}] removing replica r64/1 I190903 18:32:04.486921 35203 storage/replica_destroy.go:152 [n3,replicaGC,s3,r64/1:/Table/66/{1/6-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:04.530316 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:04.618979 35604 storage/replica_command.go:1430 [n2,replicate,s2,r65/2:/{Table/66/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r65:/{Table/66/2-Max} [(n3,s3):1, (n2,s2):2, next=3, gen=38] I190903 18:32:04.646654 35604 storage/replica_raft.go:291 [n2,s2,r65/2:/{Table/66/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:04.650907 34093 storage/queue.go:1027 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:04.667787 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:04.668024 35705 storage/store.go:2593 [n3,replicaGC,s3,r65/1:/{Table/66/2-Max}] removing replica r65/1 I190903 18:32:04.668437 35705 storage/replica_destroy.go:152 [n3,replicaGC,s3,r65/1:/{Table/66/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:04.671925 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:04.676573 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:04.681163 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:04.687190 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=66 (single col list partitioning) version=2 mtime=2019-09-03 18:32:04.685067216 +0000 UTC I190903 18:32:04.704209 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 1}] I190903 18:32:04.705436 35730 sql/lease.go:923 new lease: 66("single col list partitioning") ver=2:1567535816.432079904,0, refcount=0 I190903 18:32:04.719780 35790 storage/replica_command.go:1430 [n1,replicate,s1,r60/2:/Table/66{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r60:/Table/66{-/1} [(n3,s3):1, (n1,s1):2, next=3, gen=36] I190903 18:32:04.724992 35790 storage/replica_raft.go:291 [n1,s1,r60/2:/Table/66{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:04.736157 36127 storage/store.go:2593 [n3,replicaGC,s3,r60/1:/Table/66{-/1}] removing replica r60/1 I190903 18:32:04.736545 36127 storage/replica_destroy.go:152 [n3,replicaGC,s3,r60/1:/Table/66{-/1}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:04.752745 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=66 (single col list partitioning) version=3 mtime=2019-09-03 18:32:04.752551153 +0000 UTC I190903 18:32:04.757782 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 2}] I190903 18:32:04.766809 36213 storage/replica_command.go:1430 [n1,replicate,s1,r63/2:/Table/66/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r63:/Table/66/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=36] I190903 18:32:04.769134 36196 sql/lease.go:923 new lease: 66("single col list partitioning") ver=3:1567535822.026339950,0, refcount=0 I190903 18:32:04.770546 36213 storage/replica_raft.go:291 [n1,s1,r63/2:/Table/66/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:04.790347 34738 storage/store.go:2593 [n3,replicaGC,s3,r63/1:/Table/66/1{-/4}] removing replica r63/1 I190903 18:32:04.790738 34738 storage/replica_destroy.go:152 [n3,replicaGC,s3,r63/1:/Table/66/1{-/4}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:04.795083 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=66 (single col list partitioning) version=4 mtime=2019-09-03 18:32:04.791865447 +0000 UTC I190903 18:32:04.803048 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 66, info: {TableName:data.public."single col list partitioning" Statement:ALTER TABLE "single col list partitioning" PARTITION BY LIST (a) (PARTITION p3 VALUES IN ((3)), PARTITION p4 VALUES IN ((4))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:04.805334 36216 storage/replica_command.go:1430 [n1,s1,r64/2:/Table/66/{1/6-2}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r64:/Table/66/{1/6-2} [(n1,s1):2, next=3, gen=39] I190903 18:32:04.806825 36553 sql/lease.go:923 new lease: 66("single col list partitioning") ver=4:1567535822.027339950,0, refcount=0 I190903 18:32:04.807153 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 3}] I190903 18:32:04.808633 36705 storage/replica_command.go:1430 [n2,s2,r65/2:/{Table/66/2-Max}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r65:/{Table/66/2-Max} [(n2,s2):2, next=3, gen=39] I190903 18:32:04.809615 36216 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/66/{1/6-2}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:04.811070 36216 storage/store_snapshot.go:995 [n1,s1,r64/2:/Table/66/{1/6-2}] sending LEARNER snapshot 670f8f6e at applied index 29 I190903 18:32:04.811380 36216 storage/store_snapshot.go:1038 [n1,s1,r64/2:/Table/66/{1/6-2}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:04.812766 36771 storage/replica_raftstorage.go:793 [n3,s3,r64/3:{-}] applying LEARNER snapshot [id=670f8f6e index=29] I190903 18:32:04.816081 36705 storage/replica_raft.go:291 [n2,s2,r65/2:/{Table/66/2-Max}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:04.817282 36705 storage/store_snapshot.go:995 [n2,s2,r65/2:/{Table/66/2-Max}] sending LEARNER snapshot f1951303 at applied index 28 I190903 18:32:04.817464 36705 storage/store_snapshot.go:1038 [n2,s2,r65/2:/{Table/66/2-Max}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:04.818760 36561 storage/replica_raftstorage.go:793 [n1,s1,r65/3:{-}] applying LEARNER snapshot [id=f1951303 index=28] I190903 18:32:04.820731 36561 storage/replica_raftstorage.go:814 [n1,s1,r65/3:/{Table/66/2-Max}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=f1951303 index=28] I190903 18:32:04.822183 36771 storage/replica_raftstorage.go:814 [n3,s3,r64/3:/Table/66/{1/6-2}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=670f8f6e index=29] I190903 18:32:04.823057 36216 storage/replica_command.go:1430 [n1,s1,r64/2:/Table/66/{1/6-2}] change replicas (add [(n3,s3):3] remove []): existing descriptor r64:/Table/66/{1/6-2} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=40] I190903 18:32:04.826581 36705 storage/replica_command.go:1430 [n2,s2,r65/2:/{Table/66/2-Max}] change replicas (add [(n1,s1):3] remove []): existing descriptor r65:/{Table/66/2-Max} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=40] I190903 18:32:04.838830 36216 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/66/{1/6-2}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 18:32:04.843072 36814 storage/replica_command.go:1430 [n3,s3,r64/3:/Table/66/{1/6-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r64:/Table/66/{1/6-2} [(n1,s1):2, (n3,s3):3, next=4, gen=41] I190903 18:32:04.845646 36705 storage/replica_raft.go:291 [n2,s2,r65/2:/{Table/66/2-Max}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 E190903 18:32:04.845908 36861 storage/queue.go:1027 [n3,replicate,s3,r64/3:/Table/66/{1/6-2}] no removable replicas from range that needs a removal: [2:0, 3*:34] E190903 18:32:04.848929 36918 storage/queue.go:1027 [n1,replicate,s1,r65/3:/{Table/66/2-Max}] no removable replicas from range that needs a removal: [2:0, 3*:32] I190903 18:32:04.852578 36814 storage/replica_raft.go:291 [n3,s3,r64/3:/Table/66/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:04.852815 36950 storage/replica_command.go:1430 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r61:/Table/66/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=36] I190903 18:32:04.871553 36694 storage/replica_command.go:609 [n3,merge,s3,r62/1:/Table/66/1/{5-6}] initiating a merge of r64:/Table/66/{1/6-2} [(n3,s3):3, next=4, gen=42] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:04.872431 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 66, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:04.873659 36778 storage/store.go:2593 [n1,replicaGC,s1,r64/2:/Table/66/{1/6-2}] removing replica r64/2 I190903 18:32:04.874633 36778 storage/replica_destroy.go:152 [n1,replicaGC,s1,r64/2:/Table/66/{1/6-2}] removed 11 (0+11) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:04.876810 36950 storage/replica_raft.go:291 [n2,s2,r61/2:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:04.881744 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:04.883384 37388 storage/replica_command.go:284 [n1,split,s1,r63/2:/Table/66/1{-/4}] initiating a split of this range at key /Table/66/1/3 [r34] (zone config) I190903 18:32:04.885294 2245 storage/store.go:2593 [n3,s3,r62/1:/Table/66/1/{5-6}] removing replica r64/3 I190903 18:32:04.885305 36864 storage/store.go:2593 [n3,replicaGC,s3,r61/1:/Table/66/1/{4-5}] removing replica r61/1 I190903 18:32:04.889676 37410 storage/replica_command.go:1430 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r34:/Table/66/1/{3-4} [(n1,s1):1, next=2, gen=38] I190903 18:32:04.890446 36864 storage/replica_destroy.go:152 [n3,replicaGC,s3,r61/1:/Table/66/1/{4-5}] removed 9 (0+9) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:04.900257 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:04.900841 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:04.900884 37410 storage/replica_raft.go:291 [n1,s1,r34/1:/Table/66/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:04.903918 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:04.904701 37410 storage/store_snapshot.go:995 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] sending LEARNER snapshot 5dbc4fe9 at applied index 14 I190903 18:32:04.904907 37410 storage/store_snapshot.go:1038 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:04.906255 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:04.907666 37210 storage/replica_raftstorage.go:793 [n2,s2,r34/2:{-}] applying LEARNER snapshot [id=5dbc4fe9 index=14] I190903 18:32:04.909742 37210 storage/replica_raftstorage.go:814 [n2,s2,r34/2:/Table/66/1/{3-4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=5dbc4fe9 index=14] I190903 18:32:04.910551 37410 storage/replica_command.go:1430 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r34:/Table/66/1/{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=39] I190903 18:32:04.917451 37410 storage/replica_raft.go:291 [n1,s1,r34/1:/Table/66/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:04.919342 37410 storage/replica_command.go:1430 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r34:/Table/66/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=40] I190903 18:32:04.924097 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:04.930912 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:04.932514 37410 storage/replica_raft.go:291 [n1,s1,r34/1:/Table/66/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:04.932560 37410 storage/replica_raft.go:304 [n1,s1,r34/1:/Table/66/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:04.933837 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:04.934782 37410 storage/replica_command.go:979 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r34 failed: storage/replica_raft.go:305: [n1,s1,r34/1:/Table/66/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:04.934824 37410 storage/queue.go:1027 [n1,replicate,s1,r34/1:/Table/66/1/{3-4}] change replicas of r34 failed: storage/replica_raft.go:305: [n1,s1,r34/1:/Table/66/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:04.936248 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:04.937406 37941 storage/queue.go:1027 [n2,replicate,s2,r34/2:/Table/66/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 18:32:04.938642 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:04.989388 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:04.992815 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:04.994794 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.000230 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.004057 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.007116 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.009772 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.012652 37723 storage/replica_command.go:1430 [n1,s1,r65/3:/{Table/66/2-Max}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r65:/{Table/66/2-Max} [(n2,s2):2, (n1,s1):3, next=4, gen=41] I190903 18:32:05.012936 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.016700 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.016939 37723 storage/replica_raft.go:291 [n1,s1,r65/3:/{Table/66/2-Max}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:05.023037 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.028690 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.034656 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.038392 37698 storage/store.go:2593 [n2,replicaGC,s2,r65/2:/{Table/66/2-Max}] removing replica r65/2 I190903 18:32:05.038804 37698 storage/replica_destroy.go:152 [n2,replicaGC,s2,r65/2:/{Table/66/2-Max}] removed 11 (0+11) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:05.040158 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.043972 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.047030 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.048610 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.051360 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.055128 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.056799 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.061102 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.066780 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.069113 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.072166 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.075362 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.077072 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.080169 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.083398 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.087958 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.096851 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.098827 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.102389 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.106157 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.109391 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.112669 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.116916 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.118979 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.122119 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.128348 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.132110 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.135551 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.137877 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.140720 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.144934 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.146892 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.152500 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.157971 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.165384 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.167606 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.170870 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.182355 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:05.202589 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.204396 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.207682 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.245106 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.273552 39213 storage/replica_command.go:1430 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r62:/Table/66/{1/5-2} [(n3,s3):1, next=2, gen=43] I190903 18:32:05.282875 39183 storage/replica_command.go:1430 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r61:/Table/66/1/{4-5} [(n2,s2):2, next=3, gen=37] I190903 18:32:05.283175 39213 storage/replica_raft.go:291 [n3,s3,r62/1:/Table/66/{1/5-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:05.284295 39213 storage/store_snapshot.go:995 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] sending LEARNER snapshot 3730b89a at applied index 28 I190903 18:32:05.284531 39213 storage/store_snapshot.go:1038 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 21, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:05.285907 39273 storage/replica_raftstorage.go:793 [n1,s1,r62/2:{-}] applying LEARNER snapshot [id=3730b89a index=28] I190903 18:32:05.289859 39183 storage/replica_raft.go:291 [n2,s2,r61/2:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:05.291076 39183 storage/store_snapshot.go:995 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] sending LEARNER snapshot 7f3e6110 at applied index 33 I190903 18:32:05.291249 39273 storage/replica_raftstorage.go:814 [n1,s1,r62/2:/Table/66/{1/5-2}] applied LEARNER snapshot [total=5ms ingestion=4@5ms id=3730b89a index=28] I190903 18:32:05.291263 39183 storage/store_snapshot.go:1038 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:05.291681 39213 storage/replica_command.go:1430 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r62:/Table/66/{1/5-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=44] I190903 18:32:05.292180 39336 storage/replica_raftstorage.go:793 [n3,s3,r61/3:{-}] applying LEARNER snapshot [id=7f3e6110 index=33] I190903 18:32:05.295041 39336 storage/replica_raftstorage.go:814 [n3,s3,r61/3:/Table/66/1/{4-5}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=7f3e6110 index=33] I190903 18:32:05.295523 39183 storage/replica_command.go:1430 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] change replicas (add [(n3,s3):3] remove []): existing descriptor r61:/Table/66/1/{4-5} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=38] I190903 18:32:05.298537 39213 storage/replica_raft.go:291 [n3,s3,r62/1:/Table/66/{1/5-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:05.299215 39213 storage/replica_command.go:1430 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r62:/Table/66/{1/5-2} [(n3,s3):1, (n1,s1):2, next=3, gen=45] I190903 18:32:05.303099 39183 storage/replica_raft.go:291 [n2,s2,r61/2:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:05.303664 39183 storage/replica_command.go:1430 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r61:/Table/66/1/{4-5} [(n2,s2):2, (n3,s3):3, next=4, gen=39] I190903 18:32:05.313194 39213 storage/replica_raft.go:291 [n3,s3,r62/1:/Table/66/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:05.313239 39213 storage/replica_raft.go:304 [n3,s3,r62/1:/Table/66/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:05.314346 39213 storage/replica_command.go:979 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r62 failed: storage/replica_raft.go:305: [n3,s3,r62/1:/Table/66/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:05.314389 39213 storage/queue.go:1027 [n3,replicate,s3,r62/1:/Table/66/{1/5-2}] change replicas of r62 failed: storage/replica_raft.go:305: [n3,s3,r62/1:/Table/66/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:05.316789 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.319867 39183 storage/replica_raft.go:291 [n2,s2,r61/2:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:05.320108 39183 storage/replica_raft.go:304 [n2,s2,r61/2:/Table/66/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:05.321049 39183 storage/replica_command.go:979 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r61 failed: storage/replica_raft.go:305: [n2,s2,r61/2:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:05.321088 39183 storage/queue.go:1027 [n2,replicate,s2,r61/2:/Table/66/1/{4-5}] change replicas of r61 failed: storage/replica_raft.go:305: [n2,s2,r61/2:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:05.321497 39476 storage/replica_command.go:1430 [n2,replicate,s2,r34/2:/Table/66/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r34:/Table/66/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=40] I190903 18:32:05.333231 39476 storage/replica_raft.go:291 [n2,s2,r34/2:/Table/66/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:05.349825 39518 storage/store.go:2593 [n1,replicaGC,s1,r34/1:/Table/66/1/{3-4}] removing replica r34/1 I190903 18:32:05.350650 39518 storage/replica_destroy.go:152 [n1,replicaGC,s1,r34/1:/Table/66/1/{3-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:05.455833 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:05.458281 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.462331 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 18:32:05.562155 40118 storage/queue.go:1027 [n1,replicate,s1,r62/2:/Table/66/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:36] E190903 18:32:05.683739 40510 storage/queue.go:1027 [n3,replicate,s3,r61/3:/Table/66/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 18:32:05.734163 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:05.737446 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:05.789442 40651 storage/replica_command.go:1430 [n1,replicate,s1,r62/2:/Table/66/{1/5-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r62:/Table/66/{1/5-2} [(n3,s3):1, (n1,s1):2, next=3, gen=45] I190903 18:32:05.794245 40651 storage/replica_raft.go:291 [n1,s1,r62/2:/Table/66/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:05.807165 39482 storage/store.go:2593 [n3,replicaGC,s3,r62/1:/Table/66/{1/5-2}] removing replica r62/1 I190903 18:32:05.807541 39482 storage/replica_destroy.go:152 [n3,replicaGC,s3,r62/1:/Table/66/{1/5-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:05.878230 40914 storage/replica_command.go:1430 [n3,replicate,s3,r61/3:/Table/66/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r61:/Table/66/1/{4-5} [(n2,s2):2, (n3,s3):3, next=4, gen=39] I190903 18:32:05.894411 40914 storage/replica_raft.go:291 [n3,s3,r61/3:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:05.907946 41101 storage/store.go:2593 [n2,replicaGC,s2,r61/2:/Table/66/1/{4-5}] removing replica r61/2 I190903 18:32:05.908483 41101 storage/replica_destroy.go:152 [n2,replicaGC,s2,r61/2:/Table/66/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:06.326823 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:06.330444 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:06.332371 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:06.334213 30941 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 === RUN TestRepartitioning/multi_col_list_partitioning/multi_col_list_partitioning_-_DEFAULT I190903 18:32:06.363983 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=66 (single col list partitioning) version=5 mtime=2019-09-03 18:32:06.33747236 +0000 UTC I190903 18:32:06.381262 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 65, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning"]} I190903 18:32:06.383771 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 4}] I190903 18:32:06.436487 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=66 (single col list partitioning) version=6 mtime=2019-09-03 18:32:06.436304105 +0000 UTC I190903 18:32:06.460330 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 67, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:06.467920 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 68, info: {TableName:data.public."multi col list partitioning" Statement:CREATE TABLE "multi col list partitioning" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p56 VALUES IN ((5, 6)), PARTITION p57 VALUES IN ((5, 7))) User:root} I190903 18:32:06.470870 43384 storage/replica_command.go:284 [n1,split,s1,r65/3:/{Table/66/2-Max}] initiating a split of this range at key /Table/68 [r35] (zone config) I190903 18:32:06.481451 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:06.484112 43484 storage/replica_command.go:284 [n1,split,s1,r35/1:/{Table/68-Max}] initiating a split of this range at key /Table/68/1 [r36] (zone config) I190903 18:32:06.496538 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:06.526710 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p56 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:06.530631 44075 storage/replica_command.go:284 [n1,split,s1,r36/1:/{Table/68/1-Max}] initiating a split of this range at key /Table/68/1/3/4 [r37] (zone config) I190903 18:32:06.531239 44076 storage/replica_command.go:1430 [n1,replicate,s1,r35/1:/Table/68{-/1}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r35:/Table/68{-/1} [(n1,s1):1, next=2, gen=44] I190903 18:32:06.551721 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:06.553384 44076 storage/replica_raft.go:291 [n1,s1,r35/1:/Table/68{-/1}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:06.553647 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.554612 44076 storage/store_snapshot.go:995 [n1,replicate,s1,r35/1:/Table/68{-/1}] sending LEARNER snapshot b721a7bc at applied index 17 I190903 18:32:06.555570 44418 storage/replica_command.go:284 [n1,split,s1,r37/1:/{Table/68/1/3…-Max}] initiating a split of this range at key /Table/68/1/3/5 [r38] (zone config) I190903 18:32:06.556275 44076 storage/store_snapshot.go:1038 [n1,replicate,s1,r35/1:/Table/68{-/1}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.558327 44288 storage/replica_raftstorage.go:793 [n2,s2,r35/2:{-}] applying LEARNER snapshot [id=b721a7bc index=17] I190903 18:32:06.566165 44288 storage/replica_raftstorage.go:814 [n2,s2,r35/2:/Table/68{-/1}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=b721a7bc index=17] I190903 18:32:06.567990 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.568601 44076 storage/replica_command.go:1430 [n1,replicate,s1,r35/1:/Table/68{-/1}] change replicas (add [(n2,s2):2] remove []): existing descriptor r35:/Table/68{-/1} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=45] I190903 18:32:06.571861 44290 storage/replica_command.go:284 [n1,split,s1,r38/1:/{Table/68/1/3…-Max}] initiating a split of this range at key /Table/68/1/5/6 [r39] (zone config) I190903 18:32:06.572139 44076 storage/replica_raft.go:291 [n1,s1,r35/1:/Table/68{-/1}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:06.572813 44076 storage/replica_command.go:1430 [n1,replicate,s1,r35/1:/Table/68{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r35:/Table/68{-/1} [(n1,s1):1, (n2,s2):2, next=3, gen=46] I190903 18:32:06.573680 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.575629 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.578033 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:06.581901 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.585207 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.587802 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.588178 44076 storage/replica_raft.go:291 [n1,s1,r35/1:/Table/68{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:06.588219 44076 storage/replica_raft.go:304 [n1,s1,r35/1:/Table/68{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:06.591157 44560 storage/replica_command.go:284 [n1,split,s1,r39/1:/{Table/68/1/5…-Max}] initiating a split of this range at key /Table/68/1/5/7 [r40] (zone config) I190903 18:32:06.595271 44076 storage/replica_command.go:979 [n1,replicate,s1,r35/1:/Table/68{-/1}] could not promote [] to voter, rolling back: change replicas of r35 failed: storage/replica_raft.go:305: [n1,s1,r35/1:/Table/68{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:06.595328 44076 storage/queue.go:1027 [n1,replicate,s1,r35/1:/Table/68{-/1}] change replicas of r35 failed: storage/replica_raft.go:305: [n1,s1,r35/1:/Table/68{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:06.595629 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.595851 44515 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r37:/Table/68/1/3/{4-5} [(n1,s1):1, next=2, gen=46] I190903 18:32:06.597991 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.601517 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.605241 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.607170 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.608613 44542 storage/replica_command.go:284 [n1,split,s1,r40/1:/{Table/68/1/5…-Max}] initiating a split of this range at key /Table/68/1/5/8 [r81] (zone config) I190903 18:32:06.609357 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.610009 44515 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/68/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:06.611301 44515 storage/store_snapshot.go:995 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] sending LEARNER snapshot 88a76d2c at applied index 18 I190903 18:32:06.611536 44515 storage/store_snapshot.go:1038 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.615726 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.618237 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.618778 44435 storage/replica_raftstorage.go:793 [n2,s2,r37/2:{-}] applying LEARNER snapshot [id=88a76d2c index=18] I190903 18:32:06.620875 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.622990 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:06.628333 44435 storage/replica_raftstorage.go:814 [n2,s2,r37/2:/Table/68/1/3/{4-5}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=88a76d2c index=18] I190903 18:32:06.634302 44705 storage/replica_command.go:284 [n1,split,s1,r81/1:/{Table/68/1/5…-Max}] initiating a split of this range at key /Table/68/2 [r82] (zone config) I190903 18:32:06.642087 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:06.643623 44515 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r37:/Table/68/1/3/{4-5} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=47] I190903 18:32:06.647881 44515 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/68/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:06.659600 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.659940 44515 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r37:/Table/68/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=48] I190903 18:32:06.669127 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.671644 44515 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/68/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:06.671690 44515 storage/replica_raft.go:304 [n1,s1,r37/1:/Table/68/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:06.672574 44515 storage/replica_command.go:979 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r37 failed: storage/replica_raft.go:305: [n1,s1,r37/1:/Table/68/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:06.672613 44515 storage/queue.go:1027 [n1,replicate,s1,r37/1:/Table/68/1/3/{4-5}] change replicas of r37 failed: storage/replica_raft.go:305: [n1,s1,r37/1:/Table/68/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:06.672965 44842 storage/replica_command.go:1430 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] change replicas (add [(n3,s3):4LEARNER] remove []): existing descriptor r65:/Table/6{6/2-8} [(n1,s1):3, next=4, gen=43] I190903 18:32:06.688602 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.691261 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.695423 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:06.696204 44842 storage/replica_raft.go:291 [n1,s1,r65/3:/Table/6{6/2-8}] proposing ADD_REPLICA[(n3,s3):4LEARNER]: after=[(n1,s1):3 (n3,s3):4LEARNER] next=5 I190903 18:32:06.700361 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.701165 44842 storage/store_snapshot.go:995 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] sending LEARNER snapshot ef0938e4 at applied index 48 I190903 18:32:06.701365 44842 storage/store_snapshot.go:1038 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] streamed snapshot to (n3,s3):4LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.703159 44932 storage/replica_raftstorage.go:793 [n3,s3,r65/4:{-}] applying LEARNER snapshot [id=ef0938e4 index=48] I190903 18:32:06.703257 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.709214 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:06.711119 44932 storage/replica_raftstorage.go:814 [n3,s3,r65/4:/Table/6{6/2-8}] applied LEARNER snapshot [total=8ms ingestion=4@8ms id=ef0938e4 index=48] I190903 18:32:06.713577 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.714820 44842 storage/replica_command.go:1430 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] change replicas (add [(n3,s3):4] remove []): existing descriptor r65:/Table/6{6/2-8} [(n1,s1):3, (n3,s3):4LEARNER, next=5, gen=44] I190903 18:32:06.719855 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.722298 44842 storage/replica_raft.go:291 [n1,s1,r65/3:/Table/6{6/2-8}] proposing ADD_REPLICA[(n3,s3):4]: after=[(n1,s1):3 (n3,s3):4] next=5 I190903 18:32:06.722551 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.722860 44842 storage/replica_command.go:1430 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r65:/Table/6{6/2-8} [(n1,s1):3, (n3,s3):4, next=5, gen=45] I190903 18:32:06.727778 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.730116 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.733324 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.736445 44842 storage/replica_raft.go:291 [n1,s1,r65/3:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 E190903 18:32:06.736782 44842 storage/replica_raft.go:304 [n1,s1,r65/3:/Table/6{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 18:32:06.739397 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.739952 44842 storage/replica_command.go:979 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] could not promote [] to voter, rolling back: change replicas of r65 failed: storage/replica_raft.go:305: [n1,s1,r65/3:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) E190903 18:32:06.739992 44842 storage/queue.go:1027 [n1,replicate,s1,r65/3:/Table/6{6/2-8}] change replicas of r65 failed: storage/replica_raft.go:305: [n1,s1,r65/3:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 18:32:06.740436 45056 storage/replica_command.go:1430 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r39:/Table/68/1/5/{6-7} [(n1,s1):1, next=2, gen=48] I190903 18:32:06.742057 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.745912 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.746533 45056 storage/replica_raft.go:291 [n1,s1,r39/1:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:32:06.748139 45056 storage/store_snapshot.go:995 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] sending LEARNER snapshot 3a632224 at applied index 17 I190903 18:32:06.748311 45056 storage/store_snapshot.go:1038 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.748984 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.749446 44910 storage/replica_raftstorage.go:793 [n3,s3,r39/2:{-}] applying LEARNER snapshot [id=3a632224 index=17] I190903 18:32:06.753709 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.755781 44910 storage/replica_raftstorage.go:814 [n3,s3,r39/2:/Table/68/1/5/{6-7}] applied LEARNER snapshot [total=6ms ingestion=4@6ms id=3a632224 index=17] I190903 18:32:06.755788 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.756427 45056 storage/replica_command.go:1430 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] change replicas (add [(n3,s3):2] remove []): existing descriptor r39:/Table/68/1/5/{6-7} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=49] I190903 18:32:06.759713 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.762551 45056 storage/replica_raft.go:291 [n1,s1,r39/1:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:32:06.763339 45056 storage/replica_command.go:1430 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r39:/Table/68/1/5/{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=50] I190903 18:32:06.782017 45056 storage/replica_raft.go:291 [n1,s1,r39/1:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:06.782058 45056 storage/replica_raft.go:304 [n1,s1,r39/1:/Table/68/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:06.782957 45056 storage/replica_command.go:979 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r39 failed: storage/replica_raft.go:305: [n1,s1,r39/1:/Table/68/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:06.783009 45056 storage/queue.go:1027 [n1,replicate,s1,r39/1:/Table/68/1/5/{6-7}] change replicas of r39 failed: storage/replica_raft.go:305: [n1,s1,r39/1:/Table/68/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:06.783418 45129 storage/replica_command.go:1430 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r51:/Table/6{0/2-2} [(n1,s1):2, next=3, gen=24] I190903 18:32:06.787268 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.794621 45129 storage/replica_raft.go:291 [n1,s1,r51/2:/Table/6{0/2-2}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:06.794822 281 server/status/runtime.go:498 [n1] runtime stats: 215 MiB RSS, 979 goroutines, 61 MiB/43 MiB/138 MiB GO alloc/idle/total, 55 MiB/95 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (30x), 105 MiB/105 MiB (r/w)net I190903 18:32:06.795862 45129 storage/store_snapshot.go:995 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] sending LEARNER snapshot ba054434 at applied index 35 I190903 18:32:06.796086 45129 storage/store_snapshot.go:1038 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.797336 45181 storage/replica_raftstorage.go:793 [n3,s3,r51/3:{-}] applying LEARNER snapshot [id=ba054434 index=35] I190903 18:32:06.798743 45181 storage/replica_raftstorage.go:814 [n3,s3,r51/3:/Table/6{0/2-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ba054434 index=35] I190903 18:32:06.799253 45129 storage/replica_command.go:1430 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] change replicas (add [(n3,s3):3] remove []): existing descriptor r51:/Table/6{0/2-2} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=25] I190903 18:32:06.809727 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:06.810464 45129 storage/replica_raft.go:291 [n1,s1,r51/2:/Table/6{0/2-2}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 18:32:06.811017 45129 storage/replica_command.go:1430 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r51:/Table/6{0/2-2} [(n1,s1):2, (n3,s3):3, next=4, gen=26] I190903 18:32:06.811861 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.815780 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.818996 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:06.823793 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:06.823841 45129 storage/replica_raft.go:291 [n1,s1,r51/2:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 18:32:06.826575 45129 storage/replica_raft.go:304 [n1,s1,r51/2:/Table/6{0/2-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:06.830297 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.831087 45129 storage/replica_command.go:979 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] could not promote [] to voter, rolling back: change replicas of r51 failed: storage/replica_raft.go:305: [n1,s1,r51/2:/Table/6{0/2-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:06.831191 45129 storage/queue.go:1027 [n1,replicate,s1,r51/2:/Table/6{0/2-2}] change replicas of r51 failed: storage/replica_raft.go:305: [n1,s1,r51/2:/Table/6{0/2-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:06.835158 45212 storage/replica_command.go:1430 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r82:/{Table/68/2-Max} [(n1,s1):1, next=2, gen=50] I190903 18:32:06.837814 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.843102 45212 storage/replica_raft.go:291 [n1,s1,r82/1:/{Table/68/2-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:06.845762 45212 storage/store_snapshot.go:995 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] sending LEARNER snapshot aad0987b at applied index 15 I190903 18:32:06.845919 45212 storage/store_snapshot.go:1038 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.847473 45399 storage/replica_raftstorage.go:793 [n2,s2,r82/2:{-}] applying LEARNER snapshot [id=aad0987b index=15] I190903 18:32:06.848576 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:06.854681 45399 storage/replica_raftstorage.go:814 [n2,s2,r82/2:/{Table/68/2-Max}] applied LEARNER snapshot [total=7ms ingestion=4@7ms id=aad0987b index=15] I190903 18:32:06.855228 45212 storage/replica_command.go:1430 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r82:/{Table/68/2-Max} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=51] I190903 18:32:06.862453 45212 storage/replica_raft.go:291 [n1,s1,r82/1:/{Table/68/2-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:06.863022 45212 storage/replica_command.go:1430 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r82:/{Table/68/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=52] I190903 18:32:06.873473 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:06.907883 45212 storage/replica_raft.go:291 [n1,s1,r82/1:/{Table/68/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:06.907929 45212 storage/replica_raft.go:304 [n1,s1,r82/1:/{Table/68/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:06.908903 45212 storage/replica_command.go:979 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] could not promote [] to voter, rolling back: change replicas of r82 failed: storage/replica_raft.go:305: [n1,s1,r82/1:/{Table/68/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:06.911013 45212 storage/queue.go:1027 [n1,replicate,s1,r82/1:/{Table/68/2-Max}] change replicas of r82 failed: storage/replica_raft.go:305: [n1,s1,r82/1:/{Table/68/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:06.915497 45297 storage/queue.go:1027 [n3,replicate,s3,r65/4:/Table/6{6/2-8}] no removable replicas from range that needs a removal: [3:0, 4*:55] I190903 18:32:06.918057 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:06.919816 45299 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [(n3,s3):4LEARNER] remove []): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, next=4, gen=8] E190903 18:32:06.924085 45354 storage/queue.go:1027 [n3,replicate,s3,r51/3:/Table/6{0/2-2}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 18:32:06.925153 45299 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing ADD_REPLICA[(n3,s3):4LEARNER]: after=[(n1,s1):1 (n3,s3):4LEARNER] next=5 I190903 18:32:06.926533 45299 storage/store_snapshot.go:995 [n1,replicate,s1,r21/1:/Table/5{4-6}] sending LEARNER snapshot fef9857e at applied index 25 I190903 18:32:06.926674 45299 storage/store_snapshot.go:1038 [n1,replicate,s1,r21/1:/Table/5{4-6}] streamed snapshot to (n3,s3):4LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.927809 45066 storage/replica_raftstorage.go:793 [n3,s3,r21/4:{-}] applying LEARNER snapshot [id=fef9857e index=25] I190903 18:32:06.929053 45066 storage/replica_raftstorage.go:814 [n3,s3,r21/4:/Table/5{4-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=fef9857e index=25] I190903 18:32:06.931083 45299 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [(n3,s3):4] remove []): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n3,s3):4LEARNER, next=5, gen=9] I190903 18:32:06.944360 45299 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing ADD_REPLICA[(n3,s3):4]: after=[(n1,s1):1 (n3,s3):4] next=5 I190903 18:32:06.944957 45299 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n3,s3):4, next=5, gen=10] I190903 18:32:06.955991 45299 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):4] next=5 E190903 18:32:06.956042 45299 storage/replica_raft.go:304 [n1,s1,r21/1:/Table/5{4-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 18:32:06.958155 45299 storage/replica_command.go:979 [n1,replicate,s1,r21/1:/Table/5{4-6}] could not promote [] to voter, rolling back: change replicas of r21 failed: storage/replica_raft.go:305: [n1,s1,r21/1:/Table/5{4-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):4] next=5 to remove self (leaseholder) E190903 18:32:06.958185 45299 storage/queue.go:1027 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas of r21 failed: storage/replica_raft.go:305: [n1,s1,r21/1:/Table/5{4-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 18:32:06.958652 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:06.959587 45584 storage/replica_command.go:1430 [n1,replicate,s1,r60/2:/Table/66{-/1}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r60:/Table/66{-/1} [(n1,s1):2, next=3, gen=37] I190903 18:32:06.967344 45584 storage/replica_raft.go:291 [n1,s1,r60/2:/Table/66{-/1}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:06.968623 45584 storage/store_snapshot.go:995 [n1,replicate,s1,r60/2:/Table/66{-/1}] sending LEARNER snapshot cefc3ee5 at applied index 40 I190903 18:32:06.968778 45584 storage/store_snapshot.go:1038 [n1,replicate,s1,r60/2:/Table/66{-/1}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:06.972537 45203 storage/replica_raftstorage.go:793 [n2,s2,r60/3:{-}] applying LEARNER snapshot [id=cefc3ee5 index=40] I190903 18:32:06.973694 45203 storage/replica_raftstorage.go:814 [n2,s2,r60/3:/Table/66{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=cefc3ee5 index=40] I190903 18:32:06.977598 45584 storage/replica_command.go:1430 [n1,replicate,s1,r60/2:/Table/66{-/1}] change replicas (add [(n2,s2):3] remove []): existing descriptor r60:/Table/66{-/1} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=38] I190903 18:32:06.982535 45584 storage/replica_raft.go:291 [n1,s1,r60/2:/Table/66{-/1}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:06.983072 45584 storage/replica_command.go:1430 [n1,replicate,s1,r60/2:/Table/66{-/1}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r60:/Table/66{-/1} [(n1,s1):2, (n2,s2):3, next=4, gen=39] I190903 18:32:06.994600 45584 storage/replica_raft.go:291 [n1,s1,r60/2:/Table/66{-/1}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:06.994644 45584 storage/replica_raft.go:304 [n1,s1,r60/2:/Table/66{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:06.995554 45584 storage/replica_command.go:979 [n1,replicate,s1,r60/2:/Table/66{-/1}] could not promote [] to voter, rolling back: change replicas of r60 failed: storage/replica_raft.go:305: [n1,s1,r60/2:/Table/66{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:06.995592 45584 storage/queue.go:1027 [n1,replicate,s1,r60/2:/Table/66{-/1}] change replicas of r60 failed: storage/replica_raft.go:305: [n1,s1,r60/2:/Table/66{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:06.998873 45362 storage/queue.go:1027 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:07.029913 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:07.039708 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:07.044503 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:07.050569 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 E190903 18:32:07.057500 46024 storage/queue.go:1027 [n2,replicate,s2,r37/2:/Table/68/1/3/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:07.097286 45976 storage/replica_command.go:1430 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r39:/Table/68/1/5/{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=50] E190903 18:32:07.108760 46097 storage/queue.go:1027 [n2,replicate,s2,r82/2:/{Table/68/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:07.109754 45976 storage/replica_raft.go:291 [n3,s3,r39/2:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 18:32:07.164006 45889 storage/replica_command.go:1430 [n2,replicate,s2,r37/2:/Table/68/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r37:/Table/68/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=48] I190903 18:32:07.173183 46200 storage/replica_command.go:1430 [n3,replicate,s3,r51/3:/Table/6{0/2-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r51:/Table/6{0/2-2} [(n1,s1):2, (n3,s3):3, next=4, gen=26] I190903 18:32:07.173805 45879 storage/store.go:2593 [n1,replicaGC,s1,r39/1:/Table/68/1/5/{6-7}] removing replica r39/1 I190903 18:32:07.174805 45879 storage/replica_destroy.go:152 [n1,replicaGC,s1,r39/1:/Table/68/1/5/{6-7}] removed 9 (0+9) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:07.176256 45889 storage/replica_raft.go:291 [n2,s2,r37/2:/Table/68/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:07.178935 46242 storage/replica_command.go:1430 [n2,replicate,s2,r60/3:/Table/66{-/1}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r60:/Table/66{-/1} [(n1,s1):2, (n2,s2):3, next=4, gen=39] I190903 18:32:07.183679 46200 storage/replica_raft.go:291 [n3,s3,r51/3:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:07.185287 46425 storage/store.go:2593 [n1,replicaGC,s1,r37/1:/Table/68/1/3/{4-5}] removing replica r37/1 I190903 18:32:07.187176 46425 storage/replica_destroy.go:152 [n1,replicaGC,s1,r37/1:/Table/68/1/3/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:07.187352 46242 storage/replica_raft.go:291 [n2,s2,r60/3:/Table/66{-/1}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:07.191121 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:07.194053 46060 storage/replica_command.go:1430 [n3,replicate,s3,r21/4:/Table/5{4-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n3,s3):4, next=5, gen=10] I190903 18:32:07.197932 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:07.201360 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:07.202328 46060 storage/replica_raft.go:291 [n3,s3,r21/4:/Table/5{4-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):4] next=5 I190903 18:32:07.203397 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:07.204270 46527 storage/replica_command.go:1430 [n3,replicate,s3,r65/4:/Table/6{6/2-8}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r65:/Table/6{6/2-8} [(n1,s1):3, (n3,s3):4, next=5, gen=45] I190903 18:32:07.206014 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:07.218024 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:07.220131 46527 storage/replica_raft.go:291 [n3,s3,r65/4:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 I190903 18:32:07.224776 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=68 (multi col list partitioning - DEFAULT) version=2 mtime=2019-09-03 18:32:07.220553586 +0000 UTC I190903 18:32:07.225396 1651 server/status/runtime.go:498 [n2] runtime stats: 218 MiB RSS, 995 goroutines, 58 MiB/43 MiB/138 MiB GO alloc/idle/total, 55 MiB/96 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (31x), 105 MiB/105 MiB (r/w)net I190903 18:32:07.232199 46678 storage/replica_command.go:1430 [n2,replicate,s2,r82/2:/{Table/68/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r82:/{Table/68/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=52] I190903 18:32:07.240344 46444 storage/store.go:2593 [n1,replicaGC,s1,r51/2:/Table/6{0/2-2}] removing replica r51/2 I190903 18:32:07.243075 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 1}] I190903 18:32:07.244999 46444 storage/replica_destroy.go:152 [n1,replicaGC,s1,r51/2:/Table/6{0/2-2}] removed 8 (0+8) keys in 5ms [clear=0ms commit=4ms] I190903 18:32:07.246580 46611 storage/store.go:2593 [n1,replicaGC,s1,r60/2:/Table/66{-/1}] removing replica r60/2 I190903 18:32:07.246619 46606 sql/lease.go:923 new lease: 68("multi col list partitioning - DEFAULT") ver=2:1567535822.253650045,0, refcount=0 I190903 18:32:07.251527 46678 storage/replica_raft.go:291 [n2,s2,r82/2:/{Table/68/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:07.253086 46611 storage/replica_destroy.go:152 [n1,replicaGC,s1,r60/2:/Table/66{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:07.254321 46794 storage/replica_command.go:1430 [n2,replicate,s2,r35/2:/Table/68{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r35:/Table/68{-/1} [(n1,s1):1, (n2,s2):2, next=3, gen=46] I190903 18:32:07.263862 46794 storage/replica_raft.go:291 [n2,s2,r35/2:/Table/68{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:07.293691 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=68 (multi col list partitioning - DEFAULT) version=3 mtime=2019-09-03 18:32:07.293436748 +0000 UTC I190903 18:32:07.295567 47120 storage/store.go:2593 [n1,replicaGC,s1,r21/1:/Table/5{4-6}] removing replica r21/1 I190903 18:32:07.296007 47120 storage/replica_destroy.go:152 [n1,replicaGC,s1,r21/1:/Table/5{4-6}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:07.300297 47056 sql/lease.go:923 new lease: 68("multi col list partitioning - DEFAULT") ver=3:1567535828.533055249,0, refcount=0 I190903 18:32:07.300491 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 2}] I190903 18:32:07.334274 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=68 (multi col list partitioning - DEFAULT) version=4 mtime=2019-09-03 18:32:07.330946914 +0000 UTC I190903 18:32:07.353571 46619 storage/store.go:2593 [n1,replicaGC,s1,r65/3:/Table/6{6/2-8}] removing replica r65/3 I190903 18:32:07.355763 46619 storage/replica_destroy.go:152 [n1,replicaGC,s1,r65/3:/Table/6{6/2-8}] removed 9 (0+9) keys in 1ms [clear=0ms commit=1ms] I190903 18:32:07.356384 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 68, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:ALTER TABLE "multi col list partitioning - DEFAULT" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:07.360186 47295 sql/lease.go:923 new lease: 68("multi col list partitioning - DEFAULT") ver=4:1567535838.706658689,0, refcount=0 I190903 18:32:07.388403 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:07.399587 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:07.405071 47443 storage/store.go:2593 [n1,replicaGC,s1,r82/1:/{Table/68/2-Max}] removing replica r82/1 I190903 18:32:07.410429 47443 storage/replica_destroy.go:152 [n1,replicaGC,s1,r82/1:/{Table/68/2-Max}] removed 8 (0+8) keys in 2ms [clear=1ms commit=1ms] I190903 18:32:07.414796 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:07.416254 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:07.427886 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:07.434642 47756 storage/replica_command.go:284 [n1,split,s1,r81/1:/Table/68/{1/5/8-2}] initiating a split of this range at key /Table/68/1/5/9 [r83] (zone config) I190903 18:32:07.443172 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:07.443295 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:07.444487 48307 storage/replica_command.go:284 [n1,split,s1,r38/1:/Table/68/1/{3/5-5/6}] initiating a split of this range at key /Table/68/1/5 [r84] (zone config) I190903 18:32:07.454541 49036 storage/replica_command.go:284 [n1,split,s1,r83/1:/Table/68/{1/5/9-2}] initiating a split of this range at key /Table/68/1/6 [r85] (zone config) I190903 18:32:07.455106 48985 storage/store.go:2593 [n1,replicaGC,s1,r35/1:/Table/68{-/1}] removing replica r35/1 I190903 18:32:07.456383 48985 storage/replica_destroy.go:152 [n1,replicaGC,s1,r35/1:/Table/68{-/1}] removed 8 (0+8) keys in 1ms [clear=0ms commit=1ms] I190903 18:32:07.458615 49021 storage/replica_command.go:1430 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r84:/Table/68/1/5{-/6} [(n1,s1):1, next=2, gen=48] I190903 18:32:07.458642 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:07.460590 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.467215 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.469848 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.478961 49021 storage/replica_raft.go:291 [n1,s1,r84/1:/Table/68/1/5{-/6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:07.480247 49021 storage/store_snapshot.go:995 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] sending LEARNER snapshot 1aa696ef at applied index 14 I190903 18:32:07.480435 49021 storage/store_snapshot.go:1038 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:07.483279 49054 storage/replica_raftstorage.go:793 [n2,s2,r84/2:{-}] applying LEARNER snapshot [id=1aa696ef index=14] I190903 18:32:07.487897 2361 server/status/runtime.go:498 [n3] runtime stats: 228 MiB RSS, 1004 goroutines, 60 MiB/42 MiB/138 MiB GO alloc/idle/total, 56 MiB/98 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (32x), 105 MiB/105 MiB (r/w)net I190903 18:32:07.490693 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.495253 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.499891 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.507059 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.509909 49054 storage/replica_raftstorage.go:814 [n2,s2,r84/2:/Table/68/1/5{-/6}] applied LEARNER snapshot [total=27ms ingestion=4@26ms id=1aa696ef index=14] I190903 18:32:07.516330 49021 storage/replica_command.go:1430 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r84:/Table/68/1/5{-/6} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=49] I190903 18:32:07.517382 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.536900 49021 storage/replica_raft.go:291 [n1,s1,r84/1:/Table/68/1/5{-/6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:07.537430 49021 storage/replica_command.go:1430 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r84:/Table/68/1/5{-/6} [(n1,s1):1, (n2,s2):2, next=3, gen=50] I190903 18:32:07.539268 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.545004 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.551123 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.551240 49021 storage/replica_raft.go:291 [n1,s1,r84/1:/Table/68/1/5{-/6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:07.551275 49021 storage/replica_raft.go:304 [n1,s1,r84/1:/Table/68/1/5{-/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:07.552449 49021 storage/replica_command.go:979 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] could not promote [] to voter, rolling back: change replicas of r84 failed: storage/replica_raft.go:305: [n1,s1,r84/1:/Table/68/1/5{-/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:07.552490 49021 storage/queue.go:1027 [n1,replicate,s1,r84/1:/Table/68/1/5{-/6}] change replicas of r84 failed: storage/replica_raft.go:305: [n1,s1,r84/1:/Table/68/1/5{-/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:07.554763 49385 storage/replica_command.go:1430 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r83:/Table/68/1/{5/9-6} [(n1,s1):1, next=2, gen=52] I190903 18:32:07.559067 49385 storage/replica_raft.go:291 [n1,s1,r83/1:/Table/68/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:07.564383 48881 storage/replica_command.go:1430 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r39:/Table/68/1/5/{6-7} [(n3,s3):2, next=3, gen=51] E190903 18:32:07.567476 49462 storage/queue.go:1027 [n2,replicate,s2,r84/2:/Table/68/1/5{-/6}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 18:32:07.567914 49385 storage/store_snapshot.go:995 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] sending LEARNER snapshot bb0af78f at applied index 17 I190903 18:32:07.568121 49385 storage/store_snapshot.go:1038 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:07.569443 49481 storage/replica_raftstorage.go:793 [n2,s2,r83/2:{-}] applying LEARNER snapshot [id=bb0af78f index=17] I190903 18:32:07.576747 49481 storage/replica_raftstorage.go:814 [n2,s2,r83/2:/Table/68/1/{5/9-6}] applied LEARNER snapshot [total=7ms ingestion=4@7ms id=bb0af78f index=17] I190903 18:32:07.579012 49385 storage/replica_command.go:1430 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r83:/Table/68/1/{5/9-6} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=53] I190903 18:32:07.583238 49385 storage/replica_raft.go:291 [n1,s1,r83/1:/Table/68/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:07.584103 49385 storage/replica_command.go:1430 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r83:/Table/68/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=54] I190903 18:32:07.595981 48881 storage/replica_raft.go:291 [n3,s3,r39/2:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:32:07.597451 48881 storage/store_snapshot.go:995 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] sending LEARNER snapshot 71a1a32e at applied index 32 I190903 18:32:07.597719 48881 storage/store_snapshot.go:1038 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:07.601687 49385 storage/replica_raft.go:291 [n1,s1,r83/1:/Table/68/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:07.601724 49385 storage/replica_raft.go:304 [n1,s1,r83/1:/Table/68/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:07.603089 49512 storage/replica_raftstorage.go:793 [n2,s2,r39/3:{-}] applying LEARNER snapshot [id=71a1a32e index=32] I190903 18:32:07.606791 49385 storage/replica_command.go:979 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r83 failed: storage/replica_raft.go:305: [n1,s1,r83/1:/Table/68/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:07.606823 49385 storage/queue.go:1027 [n1,replicate,s1,r83/1:/Table/68/1/{5/9-6}] change replicas of r83 failed: storage/replica_raft.go:305: [n1,s1,r83/1:/Table/68/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:07.606971 49512 storage/replica_raftstorage.go:814 [n2,s2,r39/3:/Table/68/1/5/{6-7}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=71a1a32e index=32] I190903 18:32:07.607301 49552 storage/replica_command.go:1430 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r40:/Table/68/1/5/{7-8} [(n1,s1):1, next=2, gen=49] I190903 18:32:07.609757 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:07.612167 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:07.613840 48881 storage/replica_command.go:1430 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r39:/Table/68/1/5/{6-7} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=52] I190903 18:32:07.617451 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:07.617608 49552 storage/replica_raft.go:291 [n1,s1,r40/1:/Table/68/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:32:07.618683 49552 storage/store_snapshot.go:995 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] sending LEARNER snapshot 2af9df72 at applied index 18 I190903 18:32:07.618853 49552 storage/store_snapshot.go:1038 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:07.619680 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:07.620077 49672 storage/replica_raftstorage.go:793 [n3,s3,r40/2:{-}] applying LEARNER snapshot [id=2af9df72 index=18] I190903 18:32:07.621502 49672 storage/replica_raftstorage.go:814 [n3,s3,r40/2:/Table/68/1/5/{7-8}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=2af9df72 index=18] I190903 18:32:07.626749 48881 storage/replica_raft.go:291 [n3,s3,r39/2:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 I190903 18:32:07.627368 48881 storage/replica_command.go:1430 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r39:/Table/68/1/5/{6-7} [(n3,s3):2, (n2,s2):3, next=4, gen=53] I190903 18:32:07.627482 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.627924 49552 storage/replica_command.go:1430 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] change replicas (add [(n3,s3):2] remove []): existing descriptor r40:/Table/68/1/5/{7-8} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=50] I190903 18:32:07.659510 49552 storage/replica_raft.go:291 [n1,s1,r40/1:/Table/68/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:32:07.660360 49552 storage/replica_command.go:1430 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r40:/Table/68/1/5/{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=51] I190903 18:32:07.666801 48881 storage/replica_raft.go:291 [n3,s3,r39/2:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 E190903 18:32:07.666859 48881 storage/replica_raft.go:304 [n3,s3,r39/2:/Table/68/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:07.667709 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.673906 48881 storage/replica_command.go:979 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r39 failed: storage/replica_raft.go:305: [n3,s3,r39/2:/Table/68/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:07.673957 48881 storage/queue.go:1027 [n3,replicate,s3,r39/2:/Table/68/1/5/{6-7}] change replicas of r39 failed: storage/replica_raft.go:305: [n3,s3,r39/2:/Table/68/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:07.692313 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.693331 49552 storage/replica_raft.go:291 [n1,s1,r40/1:/Table/68/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:07.693366 49552 storage/replica_raft.go:304 [n1,s1,r40/1:/Table/68/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:07.696911 49552 storage/replica_command.go:979 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r40 failed: storage/replica_raft.go:305: [n1,s1,r40/1:/Table/68/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:07.697015 49552 storage/queue.go:1027 [n1,replicate,s1,r40/1:/Table/68/1/5/{7-8}] change replicas of r40 failed: storage/replica_raft.go:305: [n1,s1,r40/1:/Table/68/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:07.697382 49793 storage/queue.go:1027 [n2,replicate,s2,r39/3:/Table/68/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:39] I190903 18:32:07.774553 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:07.778583 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.781668 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 E190903 18:32:07.840206 50156 storage/queue.go:1027 [n2,replicate,s2,r83/2:/Table/68/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:24] E190903 18:32:07.842787 50346 storage/queue.go:1027 [n2,merge,s2,r84/2:/Table/68/1/5{-/6}] none of the remaining targets [(n1,s1):?] are legal additions to (n3,s3):2,(n2,s2):3 I190903 18:32:07.842971 50295 storage/replica_command.go:1430 [n2,replicate,s2,r84/2:/Table/68/1/5{-/6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r84:/Table/68/1/5{-/6} [(n1,s1):1, (n2,s2):2, next=3, gen=50] I190903 18:32:07.847711 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:07.849658 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.851645 50295 storage/replica_raft.go:291 [n2,s2,r84/2:/Table/68/1/5{-/6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:07.857188 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:07.859894 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.862836 50385 storage/store.go:2593 [n1,replicaGC,s1,r84/1:/Table/68/1/5{-/6}] removing replica r84/1 I190903 18:32:07.863436 50385 storage/replica_destroy.go:152 [n1,replicaGC,s1,r84/1:/Table/68/1/5{-/6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:07.865123 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:07.867122 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:07.869198 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:07.872058 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.878740 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.881959 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.889235 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:07.891090 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:07.893605 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.900654 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.922277 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.941784 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:07.948108 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 E190903 18:32:07.954257 50721 storage/queue.go:1027 [n3,replicate,s3,r40/2:/Table/68/1/5/{7-8}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:07.955228 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:07.957658 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:07.974067 50793 storage/replica_command.go:1430 [n2,replicate,s2,r83/2:/Table/68/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r83:/Table/68/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=54] I190903 18:32:07.985926 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:07.986872 50841 storage/replica_command.go:1430 [n3,replicate,s3,r40/2:/Table/68/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r40:/Table/68/1/5/{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=51] I190903 18:32:07.988862 50793 storage/replica_raft.go:291 [n2,s2,r83/2:/Table/68/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:07.989235 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:07.998052 50666 storage/store.go:2593 [n1,replicaGC,s1,r83/1:/Table/68/1/{5/9-6}] removing replica r83/1 I190903 18:32:07.998430 50666 storage/replica_destroy.go:152 [n1,replicaGC,s1,r83/1:/Table/68/1/{5/9-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:08.028747 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:08.031959 50841 storage/replica_raft.go:291 [n3,s3,r40/2:/Table/68/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 18:32:08.042904 50755 storage/store.go:2593 [n1,replicaGC,s1,r40/1:/Table/68/1/5/{7-8}] removing replica r40/1 I190903 18:32:08.043396 50755 storage/replica_destroy.go:152 [n1,replicaGC,s1,r40/1:/Table/68/1/5/{7-8}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:08.088035 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:08.095860 42273 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 === RUN TestRepartitioning/multi_col_list_partitioning_-_DEFAULT/multi_col_list_partitioning I190903 18:32:08.161214 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=68 (multi col list partitioning - DEFAULT) version=5 mtime=2019-09-03 18:32:08.104410527 +0000 UTC I190903 18:32:08.187104 51409 storage/replica_command.go:1430 [n2,replicate,s2,r39/3:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r39:/Table/68/1/5/{6-7} [(n3,s3):2, (n2,s2):3, next=4, gen=53] I190903 18:32:08.194795 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 67, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning - DEFAULT"]} I190903 18:32:08.197161 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 4}] I190903 18:32:08.232733 51409 storage/replica_raft.go:291 [n2,s2,r39/3:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 I190903 18:32:08.240428 51702 storage/replica_command.go:979 [n2,s2,r39/3:/Table/68/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r39 failed: descriptor changed: [expected] r39:/Table/68/1/5/{6-7} [(n3,s3):2, (n2,s2):3, next=4, gen=53] != [actual] r39:/Table/68/1/5/{6-7} [(n2,s2):3, next=4, gen=54] W190903 18:32:08.240586 51702 storage/replica_command.go:2033 [n2,s2,r39/3:/Table/68/1/5/{6-7}] while removing target n3,s3: change replicas of r39 failed: descriptor changed: [expected] r39:/Table/68/1/5/{6-7} [(n3,s3):2, (n2,s2):3, next=4, gen=53] != [actual] r39:/Table/68/1/5/{6-7} [(n2,s2):3, next=4, gen=54] E190903 18:32:08.240681 51702 storage/queue.go:1027 [n2,merge,s2,r84/2:/Table/68/1/5{-/6}] change replicas of r39 failed: descriptor changed: [expected] r39:/Table/68/1/5/{6-7} [(n3,s3):2, (n2,s2):3, next=4, gen=53] != [actual] r39:/Table/68/1/5/{6-7} [(n2,s2):3, next=4, gen=54] I190903 18:32:08.241835 51385 storage/store.go:2593 [n3,replicaGC,s3,r39/2:/Table/68/1/5/{6-7}] removing replica r39/2 I190903 18:32:08.242248 51385 storage/replica_destroy.go:152 [n3,replicaGC,s3,r39/2:/Table/68/1/5/{6-7}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:08.252102 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=68 (multi col list partitioning - DEFAULT) version=6 mtime=2019-09-03 18:32:08.251916639 +0000 UTC I190903 18:32:08.281019 51961 storage/replica_command.go:609 [n2,merge,s2,r84/2:/Table/68/1/5{-/6}] initiating a merge of r39:/Table/68/1/5/{6-7} [(n2,s2):3, next=4, gen=54] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:08.311699 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 69, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:08.319136 1614 storage/store.go:2593 [n2,s2,r84/2:/Table/68/1/5{-/6}] removing replica r39/3 I190903 18:32:08.340199 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 70, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:CREATE TABLE "multi col list partitioning - DEFAULT" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root} I190903 18:32:08.342248 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:08.348666 52704 storage/replica_command.go:284 [n2,split,s2,r82/2:/{Table/68/2-Max}] initiating a split of this range at key /Table/70 [r101] (zone config) I190903 18:32:08.379785 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:08.382637 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:08.383231 52847 storage/replica_command.go:284 [n2,split,s2,r101/1:/{Table/70-Max}] initiating a split of this range at key /Table/70/1 [r102] (zone config) I190903 18:32:08.455668 53189 storage/replica_command.go:284 [n2,split,s2,r102/1:/{Table/70/1-Max}] initiating a split of this range at key /Table/70/2 [r103] (zone config) I190903 18:32:08.457114 52259 storage/replica_command.go:1430 [n2,replicate,s2,r101/1:/Table/70{-/1}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r101:/Table/70{-/1} [(n2,s2):1, next=2, gen=55] I190903 18:32:08.499611 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:08.508075 52259 storage/replica_raft.go:291 [n2,s2,r101/1:/Table/70{-/1}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:08.508318 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:08.509359 52259 storage/store_snapshot.go:995 [n2,replicate,s2,r101/1:/Table/70{-/1}] sending LEARNER snapshot 2e8c52d0 at applied index 18 I190903 18:32:08.509524 52259 storage/store_snapshot.go:1038 [n2,replicate,s2,r101/1:/Table/70{-/1}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:08.512134 53553 storage/replica_command.go:284 [n2,split,s2,r102/1:/Table/70/{1-2}] initiating a split of this range at key /Table/70/1/3/4 [r104] (zone config) I190903 18:32:08.519909 53114 storage/replica_raftstorage.go:793 [n1,s1,r101/2:{-}] applying LEARNER snapshot [id=2e8c52d0 index=18] I190903 18:32:08.528152 53114 storage/replica_raftstorage.go:814 [n1,s1,r101/2:/Table/70{-/1}] applied LEARNER snapshot [total=8ms ingestion=4@8ms id=2e8c52d0 index=18] I190903 18:32:08.531651 52259 storage/replica_command.go:1430 [n2,replicate,s2,r101/1:/Table/70{-/1}] change replicas (add [(n1,s1):2] remove []): existing descriptor r101:/Table/70{-/1} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=56] I190903 18:32:08.538323 53745 storage/replica_command.go:284 [n2,split,s2,r104/1:/Table/70/{1/3/4-2}] initiating a split of this range at key /Table/70/1/3/5 [r105] (zone config) I190903 18:32:08.557506 52259 storage/replica_raft.go:291 [n2,s2,r101/1:/Table/70{-/1}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:08.561287 52259 storage/replica_command.go:1430 [n2,replicate,s2,r101/1:/Table/70{-/1}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r101:/Table/70{-/1} [(n2,s2):1, (n1,s1):2, next=3, gen=57] I190903 18:32:08.565429 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:08.575105 54140 storage/replica_command.go:284 [n2,split,s2,r105/1:/Table/70/{1/3/5-2}] initiating a split of this range at key /Table/70/1/5/7 [r106] (zone config) I190903 18:32:08.577588 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:08.578781 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:08.623540 54583 storage/replica_command.go:284 [n2,split,s2,r106/1:/Table/70/{1/5/7-2}] initiating a split of this range at key /Table/70/1/5/8 [r107] (zone config) I190903 18:32:08.623741 52259 storage/replica_raft.go:291 [n2,s2,r101/1:/Table/70{-/1}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:08.623909 52259 storage/replica_raft.go:304 [n2,s2,r101/1:/Table/70{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:08.624911 54596 storage/replica_command.go:284 [n2,split,s2,r105/1:/Table/70/1/{3/5-5/7}] initiating a split of this range at key /Table/70/1/5 [r108] (zone config) I190903 18:32:08.626058 52259 storage/replica_command.go:979 [n2,replicate,s2,r101/1:/Table/70{-/1}] could not promote [] to voter, rolling back: change replicas of r101 failed: storage/replica_raft.go:305: [n2,s2,r101/1:/Table/70{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:08.626099 52259 storage/queue.go:1027 [n2,replicate,s2,r101/1:/Table/70{-/1}] change replicas of r101 failed: storage/replica_raft.go:305: [n2,s2,r101/1:/Table/70{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:08.626572 54366 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [(n3,s3):5LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, next=5, gen=9] I190903 18:32:08.653846 54746 storage/replica_command.go:284 [n2,split,s2,r107/1:/Table/70/{1/5/8-2}] initiating a split of this range at key /Table/70/1/5/9 [r109] (zone config) I190903 18:32:08.674166 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:08.682987 54366 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):5LEARNER]: after=[(n2,s2):4 (n3,s3):5LEARNER] next=6 I190903 18:32:08.688092 54841 storage/replica_command.go:284 [n2,split,s2,r109/1:/Table/70/{1/5/9-2}] initiating a split of this range at key /Table/70/1/6 [r110] (zone config) I190903 18:32:08.690846 54366 storage/store_snapshot.go:995 [n2,replicate,s2,r4/4:/System{/tsd-tse}] sending LEARNER snapshot 729715a5 at applied index 85 I190903 18:32:08.700915 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:08.710399 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:08.747215 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:08.766287 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:08.775471 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:08.779187 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:08.792295 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:08.812142 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:08.838404 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:08.842621 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:08.848470 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:08.849514 54366 storage/store_snapshot.go:1038 [n2,replicate,s2,r4/4:/System{/tsd-tse}] streamed snapshot to (n3,s3):5LEARNER: kv pairs: 1958, log entries: 0, rate-limit: 8.0 MiB/sec, 0.16s I190903 18:32:08.853884 54861 storage/replica_raftstorage.go:793 [n3,s3,r4/5:{-}] applying LEARNER snapshot [id=729715a5 index=85] I190903 18:32:08.856794 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:08.880321 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:08.887341 54861 storage/replica_raftstorage.go:814 [n3,s3,r4/5:/System{/tsd-tse}] applied LEARNER snapshot [total=14ms ingestion=4@5ms id=729715a5 index=85] I190903 18:32:08.888154 54366 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [(n3,s3):5] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):5LEARNER, next=6, gen=10] I190903 18:32:08.889418 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:08.895518 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:08.902169 54366 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):5]: after=[(n2,s2):4 (n3,s3):5] next=6 I190903 18:32:08.902856 54366 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):4]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):5, next=6, gen=11] I190903 18:32:08.907610 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:08.914336 54366 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 E190903 18:32:08.914359 54366 storage/replica_raft.go:304 [n2,s2,r4/4:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) I190903 18:32:08.915568 54366 storage/replica_command.go:979 [n2,replicate,s2,r4/4:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/4:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) E190903 18:32:08.915599 54366 storage/queue.go:1027 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/4:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) I190903 18:32:08.918229 55555 storage/replica_command.go:1430 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r103:/{Table/70/2-Max} [(n2,s2):1, next=2, gen=56] E190903 18:32:08.926987 55594 storage/queue.go:1027 [n3,replicate,s3,r4/5:/System{/tsd-tse}] no removable replicas from range that needs a removal: [4:0, 5*:91] I190903 18:32:08.955700 55555 storage/replica_raft.go:291 [n2,s2,r103/1:/{Table/70/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:08.955846 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:08.957041 55555 storage/store_snapshot.go:995 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] sending LEARNER snapshot a4e67013 at applied index 15 I190903 18:32:08.957219 55555 storage/store_snapshot.go:1038 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:08.960615 54913 storage/replica_raftstorage.go:793 [n1,s1,r103/2:{-}] applying LEARNER snapshot [id=a4e67013 index=15] I190903 18:32:08.963705 54913 storage/replica_raftstorage.go:814 [n1,s1,r103/2:/{Table/70/2-Max}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=a4e67013 index=15] I190903 18:32:08.964941 55555 storage/replica_command.go:1430 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r103:/{Table/70/2-Max} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=57] I190903 18:32:08.967979 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:08.991788 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:08.995812 55555 storage/replica_raft.go:291 [n2,s2,r103/1:/{Table/70/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:08.996412 55555 storage/replica_command.go:1430 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r103:/{Table/70/2-Max} [(n2,s2):1, (n1,s1):2, next=3, gen=58] I190903 18:32:08.997403 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:09.002266 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:09.014750 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:09.018241 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:09.018539 55555 storage/replica_raft.go:291 [n2,s2,r103/1:/{Table/70/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:09.018572 55555 storage/replica_raft.go:304 [n2,s2,r103/1:/{Table/70/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.020620 55555 storage/replica_command.go:979 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] could not promote [] to voter, rolling back: change replicas of r103 failed: storage/replica_raft.go:305: [n2,s2,r103/1:/{Table/70/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.020691 55555 storage/queue.go:1027 [n2,replicate,s2,r103/1:/{Table/70/2-Max}] change replicas of r103 failed: storage/replica_raft.go:305: [n2,s2,r103/1:/{Table/70/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.021121 54802 storage/replica_command.go:1430 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r102:/Table/70/1{-/3/4} [(n2,s2):1, next=2, gen=57] I190903 18:32:09.025010 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:09.066650 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:09.071466 54802 storage/replica_raft.go:291 [n2,s2,r102/1:/Table/70/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:09.071775 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:09.073802 54802 storage/store_snapshot.go:995 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] sending LEARNER snapshot 5ded9c93 at applied index 21 I190903 18:32:09.074026 54802 storage/store_snapshot.go:1038 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.075730 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:09.075907 55699 storage/replica_raftstorage.go:793 [n1,s1,r102/2:{-}] applying LEARNER snapshot [id=5ded9c93 index=21] I190903 18:32:09.077251 55699 storage/replica_raftstorage.go:814 [n1,s1,r102/2:/Table/70/1{-/3/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5ded9c93 index=21] I190903 18:32:09.077881 54802 storage/replica_command.go:1430 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r102:/Table/70/1{-/3/4} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=58] I190903 18:32:09.084469 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:09.088414 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:09.093076 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:09.095946 54802 storage/replica_raft.go:291 [n2,s2,r102/1:/Table/70/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:09.097424 54802 storage/replica_command.go:1430 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r102:/Table/70/1{-/3/4} [(n2,s2):1, (n1,s1):2, next=3, gen=59] I190903 18:32:09.100702 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:09.110695 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:09.113321 54802 storage/replica_raft.go:291 [n2,s2,r102/1:/Table/70/1{-/3/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:09.113359 54802 storage/replica_raft.go:304 [n2,s2,r102/1:/Table/70/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.116207 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:09.118216 54802 storage/replica_command.go:979 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r102 failed: storage/replica_raft.go:305: [n2,s2,r102/1:/Table/70/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.118245 54802 storage/queue.go:1027 [n2,replicate,s2,r102/1:/Table/70/1{-/3/4}] change replicas of r102 failed: storage/replica_raft.go:305: [n2,s2,r102/1:/Table/70/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.122475 56387 storage/queue.go:1027 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:23] I190903 18:32:09.123444 56322 storage/replica_command.go:1430 [n2,replicate,s2,r47/2:/Table/60{-/1}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r47:/Table/60{-/1} [(n2,s2):2, next=3, gen=20] I190903 18:32:09.134803 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 E190903 18:32:09.135124 56479 storage/queue.go:1027 [n1,replicate,s1,r102/2:/Table/70/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 18:32:09.143852 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:09.145293 56322 storage/replica_raft.go:291 [n2,s2,r47/2:/Table/60{-/1}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:09.147524 56322 storage/store_snapshot.go:995 [n2,replicate,s2,r47/2:/Table/60{-/1}] sending LEARNER snapshot e992c97e at applied index 41 I190903 18:32:09.148082 56322 storage/store_snapshot.go:1038 [n2,replicate,s2,r47/2:/Table/60{-/1}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.165451 56612 storage/replica_raftstorage.go:793 [n3,s3,r47/3:{-}] applying LEARNER snapshot [id=e992c97e index=41] I190903 18:32:09.176830 56612 storage/replica_raftstorage.go:814 [n3,s3,r47/3:/Table/60{-/1}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=e992c97e index=41] I190903 18:32:09.177951 56322 storage/replica_command.go:1430 [n2,replicate,s2,r47/2:/Table/60{-/1}] change replicas (add [(n3,s3):3] remove []): existing descriptor r47:/Table/60{-/1} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=21] I190903 18:32:09.186910 56322 storage/replica_raft.go:291 [n2,s2,r47/2:/Table/60{-/1}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:09.187448 56322 storage/replica_command.go:1430 [n2,replicate,s2,r47/2:/Table/60{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r47:/Table/60{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=22] I190903 18:32:09.227879 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:09.236531 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:09.244570 56322 storage/replica_raft.go:291 [n2,s2,r47/2:/Table/60{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:09.244614 56322 storage/replica_raft.go:304 [n2,s2,r47/2:/Table/60{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:09.245468 56322 storage/replica_command.go:979 [n2,replicate,s2,r47/2:/Table/60{-/1}] could not promote [] to voter, rolling back: change replicas of r47 failed: storage/replica_raft.go:305: [n2,s2,r47/2:/Table/60{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:09.245509 56322 storage/queue.go:1027 [n2,replicate,s2,r47/2:/Table/60{-/1}] change replicas of r47 failed: storage/replica_raft.go:305: [n2,s2,r47/2:/Table/60{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:09.247068 56872 storage/replica_command.go:1430 [n2,replicate,s2,r60/3:/Table/66{-/1}] change replicas (add [(n3,s3):4LEARNER] remove []): existing descriptor r60:/Table/66{-/1} [(n2,s2):3, next=4, gen=40] E190903 18:32:09.247727 56454 storage/queue.go:1027 [n1,replicate,s1,r101/2:/Table/70{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 18:32:09.258237 56872 storage/replica_raft.go:291 [n2,s2,r60/3:/Table/66{-/1}] proposing ADD_REPLICA[(n3,s3):4LEARNER]: after=[(n2,s2):3 (n3,s3):4LEARNER] next=5 I190903 18:32:09.263602 56872 storage/store_snapshot.go:995 [n2,replicate,s2,r60/3:/Table/66{-/1}] sending LEARNER snapshot 0efec821 at applied index 54 I190903 18:32:09.263796 56872 storage/store_snapshot.go:1038 [n2,replicate,s2,r60/3:/Table/66{-/1}] streamed snapshot to (n3,s3):4LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.266231 56891 storage/replica_raftstorage.go:793 [n3,s3,r60/4:{-}] applying LEARNER snapshot [id=0efec821 index=54] I190903 18:32:09.267148 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:09.268109 56891 storage/replica_raftstorage.go:814 [n3,s3,r60/4:/Table/66{-/1}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=0efec821 index=54] I190903 18:32:09.268775 56872 storage/replica_command.go:1430 [n2,replicate,s2,r60/3:/Table/66{-/1}] change replicas (add [(n3,s3):4] remove []): existing descriptor r60:/Table/66{-/1} [(n2,s2):3, (n3,s3):4LEARNER, next=5, gen=41] I190903 18:32:09.269768 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:09.274309 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:09.284918 56872 storage/replica_raft.go:291 [n2,s2,r60/3:/Table/66{-/1}] proposing ADD_REPLICA[(n3,s3):4]: after=[(n2,s2):3 (n3,s3):4] next=5 I190903 18:32:09.285841 56872 storage/replica_command.go:1430 [n2,replicate,s2,r60/3:/Table/66{-/1}] change replicas (add [] remove [(n2,s2):3]): existing descriptor r60:/Table/66{-/1} [(n2,s2):3, (n3,s3):4, next=5, gen=42] I190903 18:32:09.288547 57039 storage/replica_command.go:1430 [n1,replicate,s1,r102/2:/Table/70/1{-/3/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r102:/Table/70/1{-/3/4} [(n2,s2):1, (n1,s1):2, next=3, gen=59] I190903 18:32:09.298345 57039 storage/replica_raft.go:291 [n1,s1,r102/2:/Table/70/1{-/3/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:09.299005 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:09.300951 56872 storage/replica_raft.go:291 [n2,s2,r60/3:/Table/66{-/1}] proposing REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 E190903 18:32:09.302207 56872 storage/replica_raft.go:304 [n2,s2,r60/3:/Table/66{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 18:32:09.305916 56872 storage/replica_command.go:979 [n2,replicate,s2,r60/3:/Table/66{-/1}] could not promote [] to voter, rolling back: change replicas of r60 failed: storage/replica_raft.go:305: [n2,s2,r60/3:/Table/66{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) E190903 18:32:09.307686 56872 storage/queue.go:1027 [n2,replicate,s2,r60/3:/Table/66{-/1}] change replicas of r60 failed: storage/replica_raft.go:305: [n2,s2,r60/3:/Table/66{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 18:32:09.308522 57008 storage/replica_command.go:1430 [n2,replicate,s2,r82/2:/Table/{68/2-70}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r82:/Table/{68/2-70} [(n2,s2):2, next=3, gen=54] I190903 18:32:09.310190 57150 storage/replica_command.go:1430 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):4]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):5, next=6, gen=11] I190903 18:32:09.319714 57005 storage/store.go:2593 [n2,replicaGC,s2,r102/1:/Table/70/1{-/3/4}] removing replica r102/1 I190903 18:32:09.320087 57005 storage/replica_destroy.go:152 [n2,replicaGC,s2,r102/1:/Table/70/1{-/3/4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:09.330275 57008 storage/replica_raft.go:291 [n2,s2,r82/2:/Table/{68/2-70}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:09.332082 57008 storage/store_snapshot.go:995 [n2,replicate,s2,r82/2:/Table/{68/2-70}] sending LEARNER snapshot 2d5ebe7a at applied index 37 I190903 18:32:09.332318 57008 storage/store_snapshot.go:1038 [n2,replicate,s2,r82/2:/Table/{68/2-70}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.335876 57150 storage/replica_raft.go:291 [n3,s3,r4/5:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 I190903 18:32:09.341210 57127 storage/replica_raftstorage.go:793 [n1,s1,r82/3:{-}] applying LEARNER snapshot [id=2d5ebe7a index=37] I190903 18:32:09.345737 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:09.350121 57180 storage/store.go:2593 [n2,replicaGC,s2,r4/4:/System{/tsd-tse}] removing replica r4/4 I190903 18:32:09.351092 57180 storage/replica_destroy.go:152 [n2,replicaGC,s2,r4/4:/System{/tsd-tse}] removed 1950 (1941+9) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:09.355818 57127 storage/replica_raftstorage.go:814 [n1,s1,r82/3:/Table/{68/2-70}] applied LEARNER snapshot [total=15ms ingestion=4@14ms id=2d5ebe7a index=37] I190903 18:32:09.356825 57008 storage/replica_command.go:1430 [n2,replicate,s2,r82/2:/Table/{68/2-70}] change replicas (add [(n1,s1):3] remove []): existing descriptor r82:/Table/{68/2-70} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=55] I190903 18:32:09.367500 57008 storage/replica_raft.go:291 [n2,s2,r82/2:/Table/{68/2-70}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:09.368130 57008 storage/replica_command.go:1430 [n2,replicate,s2,r82/2:/Table/{68/2-70}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r82:/Table/{68/2-70} [(n2,s2):2, (n1,s1):3, next=4, gen=56] I190903 18:32:09.381863 57008 storage/replica_raft.go:291 [n2,s2,r82/2:/Table/{68/2-70}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:09.381910 57008 storage/replica_raft.go:304 [n2,s2,r82/2:/Table/{68/2-70}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:09.382778 57008 storage/replica_command.go:979 [n2,replicate,s2,r82/2:/Table/{68/2-70}] could not promote [] to voter, rolling back: change replicas of r82 failed: storage/replica_raft.go:305: [n2,s2,r82/2:/Table/{68/2-70}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:09.382812 57008 storage/queue.go:1027 [n2,replicate,s2,r82/2:/Table/{68/2-70}] change replicas of r82 failed: storage/replica_raft.go:305: [n2,s2,r82/2:/Table/{68/2-70}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:09.383212 57399 storage/replica_command.go:1430 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r106:/Table/70/1/5/{7-8} [(n2,s2):1, next=2, gen=60] I190903 18:32:09.399501 57399 storage/replica_raft.go:291 [n2,s2,r106/1:/Table/70/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:09.401337 57399 storage/store_snapshot.go:995 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] sending LEARNER snapshot 33fc3da3 at applied index 21 I190903 18:32:09.401492 57399 storage/store_snapshot.go:1038 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.402775 57479 storage/replica_raftstorage.go:793 [n3,s3,r106/2:{-}] applying LEARNER snapshot [id=33fc3da3 index=21] I190903 18:32:09.416987 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:09.424090 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:09.424823 57479 storage/replica_raftstorage.go:814 [n3,s3,r106/2:/Table/70/1/5/{7-8}] applied LEARNER snapshot [total=22ms ingestion=4@22ms id=33fc3da3 index=21] I190903 18:32:09.425473 57399 storage/replica_command.go:1430 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] change replicas (add [(n3,s3):2] remove []): existing descriptor r106:/Table/70/1/5/{7-8} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=61] I190903 18:32:09.435258 212 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:09.445410 57399 storage/replica_raft.go:291 [n2,s2,r106/1:/Table/70/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 18:32:09.445977 57399 storage/replica_command.go:1430 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r106:/Table/70/1/5/{7-8} [(n2,s2):1, (n3,s3):2, next=3, gen=62] I190903 18:32:09.469228 57399 storage/replica_raft.go:291 [n2,s2,r106/1:/Table/70/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 18:32:09.469278 57399 storage/replica_raft.go:304 [n2,s2,r106/1:/Table/70/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:09.470166 57399 storage/replica_command.go:979 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r106 failed: storage/replica_raft.go:305: [n2,s2,r106/1:/Table/70/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:09.470231 57399 storage/queue.go:1027 [n2,replicate,s2,r106/1:/Table/70/1/5/{7-8}] change replicas of r106 failed: storage/replica_raft.go:305: [n2,s2,r106/1:/Table/70/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:09.472438 57728 storage/replica_command.go:1430 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r105:/Table/70/1/{3/5-5} [(n2,s2):1, next=2, gen=60] E190903 18:32:09.475092 57448 storage/queue.go:1027 [n3,replicate,s3,r47/3:/Table/60{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:49] I190903 18:32:09.484191 57728 storage/replica_raft.go:291 [n2,s2,r105/1:/Table/70/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:09.486626 57728 storage/store_snapshot.go:995 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] sending LEARNER snapshot 605b9d70 at applied index 25 I190903 18:32:09.486820 57728 storage/store_snapshot.go:1038 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.489950 57818 storage/replica_raftstorage.go:793 [n1,s1,r105/2:{-}] applying LEARNER snapshot [id=605b9d70 index=25] I190903 18:32:09.494839 57818 storage/replica_raftstorage.go:814 [n1,s1,r105/2:/Table/70/1/{3/5-5}] applied LEARNER snapshot [total=5ms ingestion=4@5ms id=605b9d70 index=25] I190903 18:32:09.495774 57634 storage/replica_command.go:1430 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r103:/{Table/70/2-Max} [(n2,s2):1, (n1,s1):2, next=3, gen=58] I190903 18:32:09.497438 57728 storage/replica_command.go:1430 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] change replicas (add [(n1,s1):2] remove []): existing descriptor r105:/Table/70/1/{3/5-5} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=61] I190903 18:32:09.501293 57634 storage/replica_raft.go:291 [n1,s1,r103/2:/{Table/70/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:09.513871 57819 storage/store.go:2593 [n2,replicaGC,s2,r103/1:/{Table/70/2-Max}] removing replica r103/1 I190903 18:32:09.514386 57819 storage/replica_destroy.go:152 [n2,replicaGC,s2,r103/1:/{Table/70/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:09.522595 57728 storage/replica_raft.go:291 [n2,s2,r105/1:/Table/70/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:09.523459 57728 storage/replica_command.go:1430 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r105:/Table/70/1/{3/5-5} [(n2,s2):1, (n1,s1):2, next=3, gen=62] I190903 18:32:09.575715 58093 storage/replica_command.go:1430 [n1,replicate,s1,r101/2:/Table/70{-/1}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r101:/Table/70{-/1} [(n2,s2):1, (n1,s1):2, next=3, gen=57] I190903 18:32:09.577374 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:09.582301 58093 storage/replica_raft.go:291 [n1,s1,r101/2:/Table/70{-/1}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:09.582302 57728 storage/replica_raft.go:291 [n2,s2,r105/1:/Table/70/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:09.582557 57728 storage/replica_raft.go:304 [n2,s2,r105/1:/Table/70/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.583876 57728 storage/replica_command.go:979 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r105 failed: storage/replica_raft.go:305: [n2,s2,r105/1:/Table/70/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.583949 57728 storage/queue.go:1027 [n2,replicate,s2,r105/1:/Table/70/1/{3/5-5}] change replicas of r105 failed: storage/replica_raft.go:305: [n2,s2,r105/1:/Table/70/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.584456 57799 storage/replica_command.go:1430 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r107:/Table/70/1/5/{8-9} [(n2,s2):1, next=2, gen=61] I190903 18:32:09.593328 57875 storage/store.go:2593 [n2,replicaGC,s2,r101/1:/Table/70{-/1}] removing replica r101/1 I190903 18:32:09.593678 57875 storage/replica_destroy.go:152 [n2,replicaGC,s2,r101/1:/Table/70{-/1}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:09.594322 57799 storage/replica_raft.go:291 [n2,s2,r107/1:/Table/70/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:09.597183 57799 storage/store_snapshot.go:995 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] sending LEARNER snapshot 933c3e9b at applied index 20 I190903 18:32:09.597353 57799 storage/store_snapshot.go:1038 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.598541 56908 storage/replica_raftstorage.go:793 [n1,s1,r107/2:{-}] applying LEARNER snapshot [id=933c3e9b index=20] I190903 18:32:09.599780 56908 storage/replica_raftstorage.go:814 [n1,s1,r107/2:/Table/70/1/5/{8-9}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=933c3e9b index=20] I190903 18:32:09.600986 57799 storage/replica_command.go:1430 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] change replicas (add [(n1,s1):2] remove []): existing descriptor r107:/Table/70/1/5/{8-9} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=62] I190903 18:32:09.610404 57799 storage/replica_raft.go:291 [n2,s2,r107/1:/Table/70/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:09.611060 57799 storage/replica_command.go:1430 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r107:/Table/70/1/5/{8-9} [(n2,s2):1, (n1,s1):2, next=3, gen=63] I190903 18:32:09.627559 57799 storage/replica_raft.go:291 [n2,s2,r107/1:/Table/70/1/5/{8-9}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:09.627628 57799 storage/replica_raft.go:304 [n2,s2,r107/1:/Table/70/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.630143 57799 storage/replica_command.go:979 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r107 failed: storage/replica_raft.go:305: [n2,s2,r107/1:/Table/70/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.630171 57799 storage/queue.go:1027 [n2,replicate,s2,r107/1:/Table/70/1/5/{8-9}] change replicas of r107 failed: storage/replica_raft.go:305: [n2,s2,r107/1:/Table/70/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.632844 58326 storage/replica_command.go:1430 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r110:/Table/70/{1/6-2} [(n2,s2):1, next=2, gen=62] E190903 18:32:09.635291 58331 storage/queue.go:1027 [n3,replicate,s3,r60/4:/Table/66{-/1}] no removable replicas from range that needs a removal: [3:0, 4*:61] I190903 18:32:09.649118 58326 storage/replica_raft.go:291 [n2,s2,r110/1:/Table/70/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:09.653108 58326 storage/store_snapshot.go:995 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] sending LEARNER snapshot 7c5b6fb5 at applied index 18 I190903 18:32:09.653317 58326 storage/store_snapshot.go:1038 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:09.654643 58267 storage/replica_raftstorage.go:793 [n1,s1,r110/2:{-}] applying LEARNER snapshot [id=7c5b6fb5 index=18] I190903 18:32:09.684387 58267 storage/replica_raftstorage.go:814 [n1,s1,r110/2:/Table/70/{1/6-2}] applied LEARNER snapshot [total=30ms ingestion=4@29ms id=7c5b6fb5 index=18] I190903 18:32:09.685923 58326 storage/replica_command.go:1430 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r110:/Table/70/{1/6-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=63] I190903 18:32:09.695888 58326 storage/replica_raft.go:291 [n2,s2,r110/1:/Table/70/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:09.696884 58326 storage/replica_command.go:1430 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r110:/Table/70/{1/6-2} [(n2,s2):1, (n1,s1):2, next=3, gen=64] I190903 18:32:09.720776 58326 storage/replica_raft.go:291 [n2,s2,r110/1:/Table/70/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:09.720825 58326 storage/replica_raft.go:304 [n2,s2,r110/1:/Table/70/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:09.721594 58326 storage/replica_command.go:979 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r110 failed: storage/replica_raft.go:305: [n2,s2,r110/1:/Table/70/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.721683 58326 storage/queue.go:1027 [n2,replicate,s2,r110/1:/Table/70/{1/6-2}] change replicas of r110 failed: storage/replica_raft.go:305: [n2,s2,r110/1:/Table/70/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:09.724972 58126 storage/queue.go:1027 [n1,replicate,s1,r82/3:/Table/{68/2-70}] no removable replicas from range that needs a removal: [2:0, 3*:45] E190903 18:32:09.726426 58128 storage/queue.go:1027 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] no removable replicas from range that needs a removal: [1:0, 2*:29] E190903 18:32:09.736035 58510 storage/queue.go:1027 [n1,replicate,s1,r110/2:/Table/70/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:09.860649 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:09.883492 58782 storage/replica_command.go:1430 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r106:/Table/70/1/5/{7-8} [(n2,s2):1, (n3,s3):2, next=3, gen=62] I190903 18:32:09.891443 58782 storage/replica_raft.go:291 [n3,s3,r106/2:/Table/70/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 18:32:09.935433 58985 storage/replica_command.go:1430 [n1,replicate,s1,r82/3:/Table/{68/2-70}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r82:/Table/{68/2-70} [(n2,s2):2, (n1,s1):3, next=4, gen=56] I190903 18:32:09.939647 58985 storage/replica_raft.go:291 [n1,s1,r82/3:/Table/{68/2-70}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:09.949285 58545 storage/replica_command.go:1430 [n3,replicate,s3,r47/3:/Table/60{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r47:/Table/60{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=22] I190903 18:32:09.951336 59044 storage/store.go:2593 [n2,replicaGC,s2,r106/1:/Table/70/1/5/{7-8}] removing replica r106/1 I190903 18:32:09.951757 59044 storage/replica_destroy.go:152 [n2,replicaGC,s2,r106/1:/Table/70/1/5/{7-8}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:09.952623 59088 storage/store.go:2593 [n2,replicaGC,s2,r82/2:/Table/{68/2-70}] removing replica r82/2 I190903 18:32:09.953107 59088 storage/replica_destroy.go:152 [n2,replicaGC,s2,r82/2:/Table/{68/2-70}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:09.958371 58545 storage/replica_raft.go:291 [n3,s3,r47/3:/Table/60{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:09.997499 58992 storage/replica_command.go:1430 [n1,replicate,s1,r105/2:/Table/70/1/{3/5-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r105:/Table/70/1/{3/5-5} [(n2,s2):1, (n1,s1):2, next=3, gen=62] I190903 18:32:10.006954 59236 storage/replica_command.go:1430 [n3,replicate,s3,r60/4:/Table/66{-/1}] change replicas (add [] remove [(n2,s2):3]): existing descriptor r60:/Table/66{-/1} [(n2,s2):3, (n3,s3):4, next=5, gen=42] I190903 18:32:10.008348 59412 storage/store.go:2593 [n2,replicaGC,s2,r47/2:/Table/60{-/1}] removing replica r47/2 I190903 18:32:10.008816 59412 storage/replica_destroy.go:152 [n2,replicaGC,s2,r47/2:/Table/60{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:10.012834 58992 storage/replica_raft.go:291 [n1,s1,r105/2:/Table/70/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:10.015897 59091 storage/replica_command.go:1430 [n1,replicate,s1,r107/2:/Table/70/1/5/{8-9}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r107:/Table/70/1/5/{8-9} [(n2,s2):1, (n1,s1):2, next=3, gen=63] I190903 18:32:10.019975 59091 storage/replica_raft.go:291 [n1,s1,r107/2:/Table/70/1/5/{8-9}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:10.021881 59236 storage/replica_raft.go:291 [n3,s3,r60/4:/Table/66{-/1}] proposing REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 I190903 18:32:10.027318 59493 storage/store.go:2593 [n2,replicaGC,s2,r105/1:/Table/70/1/{3/5-5}] removing replica r105/1 I190903 18:32:10.027729 59493 storage/replica_destroy.go:152 [n2,replicaGC,s2,r105/1:/Table/70/1/{3/5-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:10.028521 58643 storage/store.go:2593 [n2,replicaGC,s2,r107/1:/Table/70/1/5/{8-9}] removing replica r107/1 I190903 18:32:10.028876 58643 storage/replica_destroy.go:152 [n2,replicaGC,s2,r107/1:/Table/70/1/5/{8-9}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:10.079203 59590 storage/store.go:2593 [n2,replicaGC,s2,r60/3:/Table/66{-/1}] removing replica r60/3 I190903 18:32:10.079618 59590 storage/replica_destroy.go:152 [n2,replicaGC,s2,r60/3:/Table/66{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:10.133754 59689 storage/replica_command.go:1430 [n1,replicate,s1,r110/2:/Table/70/{1/6-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r110:/Table/70/{1/6-2} [(n2,s2):1, (n1,s1):2, next=3, gen=64] I190903 18:32:10.137597 59689 storage/replica_raft.go:291 [n1,s1,r110/2:/Table/70/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:10.146386 59846 storage/store.go:2593 [n2,replicaGC,s2,r110/1:/Table/70/{1/6-2}] removing replica r110/1 I190903 18:32:10.146739 59846 storage/replica_destroy.go:152 [n2,replicaGC,s2,r110/1:/Table/70/{1/6-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:10.401416 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:10.405522 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:10.408644 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:10.413887 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:10.416074 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:10.418117 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:10.428376 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:10.435924 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:10.441247 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=70 (multi col list partitioning) version=2 mtime=2019-09-03 18:32:10.439676325 +0000 UTC I190903 18:32:10.448348 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 1}] I190903 18:32:10.451570 60639 sql/lease.go:923 new lease: 70("multi col list partitioning") ver=2:1567535831.158526359,0, refcount=0 I190903 18:32:10.506623 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=70 (multi col list partitioning) version=3 mtime=2019-09-03 18:32:10.50641074 +0000 UTC I190903 18:32:10.511717 60844 sql/lease.go:923 new lease: 70("multi col list partitioning") ver=3:1567535837.098318411,0, refcount=0 I190903 18:32:10.516089 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 2}] I190903 18:32:10.549538 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=70 (multi col list partitioning) version=4 mtime=2019-09-03 18:32:10.547171505 +0000 UTC I190903 18:32:10.556971 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 70, info: {TableName:data.public."multi col list partitioning" Statement:ALTER TABLE "multi col list partitioning" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p56 VALUES IN ((5, 6)), PARTITION p57 VALUES IN ((5, 7))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:10.559201 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 3}] I190903 18:32:10.561081 61613 storage/replica_command.go:1430 [n2,s2,r109/1:/Table/70/1/{5/9-6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r109:/Table/70/1/{5/9-6} [(n2,s2):1, next=2, gen=62] I190903 18:32:10.562383 61419 sql/lease.go:923 new lease: 70("multi col list partitioning") ver=4:1567535837.099318411,0, refcount=0 I190903 18:32:10.571505 61613 storage/replica_raft.go:291 [n2,s2,r109/1:/Table/70/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:10.591778 61613 storage/store_snapshot.go:995 [n2,s2,r109/1:/Table/70/1/{5/9-6}] sending LEARNER snapshot 99f96b2a at applied index 23 I190903 18:32:10.592125 61613 storage/store_snapshot.go:1038 [n2,s2,r109/1:/Table/70/1/{5/9-6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 18:32:10.594217 61717 storage/replica_raftstorage.go:793 [n1,s1,r109/2:{-}] applying LEARNER snapshot [id=99f96b2a index=23] I190903 18:32:10.602065 61717 storage/replica_raftstorage.go:814 [n1,s1,r109/2:/Table/70/1/{5/9-6}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=99f96b2a index=23] I190903 18:32:10.602868 61613 storage/replica_command.go:1430 [n2,s2,r109/1:/Table/70/1/{5/9-6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r109:/Table/70/1/{5/9-6} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=63] E190903 18:32:10.609980 61612 storage/queue.go:1027 [n2,merge,s2,r109/1:/Table/70/1/{5/9-6}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):2 I190903 18:32:10.613690 61613 storage/replica_raft.go:291 [n2,s2,r109/1:/Table/70/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 E190903 18:32:10.618471 61827 storage/queue.go:1027 [n1,replicate,s1,r109/2:/Table/70/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:10.722147 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:10.732678 61909 storage/replica_command.go:1430 [n1,s1,r109/2:/Table/70/1/{5/9-6}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r109:/Table/70/1/{5/9-6} [(n2,s2):1, (n1,s1):2, next=3, gen=64] I190903 18:32:10.734142 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:10.742927 62646 storage/replica_command.go:1430 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r108:/Table/70/1/5{-/7} [(n2,s2):1, next=2, gen=60] I190903 18:32:10.757631 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p56 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:10.759536 62806 storage/replica_command.go:284 [n2,split,s2,r108/1:/Table/70/1/5{-/7}] initiating a split of this range at key /Table/70/1/5/6 [r111] (zone config) I190903 18:32:10.760803 62646 storage/replica_raft.go:291 [n2,s2,r108/1:/Table/70/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:10.761870 62646 storage/store_snapshot.go:995 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] sending LEARNER snapshot e2904623 at applied index 17 I190903 18:32:10.762090 62646 storage/store_snapshot.go:1038 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:10.764417 62099 storage/replica_raftstorage.go:793 [n1,s1,r108/2:{-}] applying LEARNER snapshot [id=e2904623 index=17] I190903 18:32:10.765759 62099 storage/replica_raftstorage.go:814 [n1,s1,r108/2:/Table/70/1/5{-/7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e2904623 index=17] I190903 18:32:10.766600 62646 storage/replica_command.go:1430 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] change replicas (add [(n1,s1):2] remove []): existing descriptor r108:/Table/70/1/5{-/7} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=61] I190903 18:32:10.767706 62806 storage/split_queue.go:149 [n2,split,s2,r108/1:/Table/70/1/5{-/7}] split saw concurrent descriptor modification; maybe retrying I190903 18:32:10.767872 62905 storage/replica_command.go:284 [n2,split,s2,r108/1:/Table/70/1/5{-/7}] initiating a split of this range at key /Table/70/1/5/6 [r112] (zone config); r108/2 is being probed (may or may not need a Raft snapshot) I190903 18:32:10.772876 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:10.773249 61909 storage/replica_raft.go:291 [n1,s1,r109/2:/Table/70/1/{5/9-6}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:10.776327 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:10.793322 62646 storage/replica_raft.go:291 [n2,s2,r108/1:/Table/70/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:10.793932 62646 storage/replica_command.go:1430 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r108:/Table/70/1/5{-/7} [(n2,s2):1, (n1,s1):2, next=3, gen=62] I190903 18:32:10.797227 61271 storage/replica_command.go:609 [n1,merge,s1,r107/2:/Table/70/1/5/{8-9}] initiating a merge of r109:/Table/70/1/{5/9-6} [(n1,s1):2, next=3, gen=65] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:10.806795 61795 storage/store.go:2593 [n2,replicaGC,s2,r109/1:/Table/70/1/{5/9-6}] removing replica r109/1 I190903 18:32:10.807012 62905 storage/split_queue.go:149 [n2,split,s2,r108/1:/Table/70/1/5{-/7}] split saw concurrent descriptor modification; maybe retrying I190903 18:32:10.807184 63366 storage/replica_command.go:284 [n2,split,s2,r108/1:/Table/70/1/5{-/7}] initiating a split of this range at key /Table/70/1/5/6 [r113] (zone config) I190903 18:32:10.807234 61795 storage/replica_destroy.go:152 [n2,replicaGC,s2,r109/1:/Table/70/1/{5/9-6}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:10.812925 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:10.813066 62646 storage/replica_raft.go:291 [n2,s2,r108/1:/Table/70/1/5{-/7}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:10.813452 62646 storage/replica_raft.go:304 [n2,s2,r108/1:/Table/70/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:10.815032 62646 storage/replica_command.go:979 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r108 failed: storage/replica_raft.go:305: [n2,s2,r108/1:/Table/70/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:10.815057 62646 storage/queue.go:1027 [n2,replicate,s2,r108/1:/Table/70/1/5{-/7}] change replicas of r108 failed: storage/replica_raft.go:305: [n2,s2,r108/1:/Table/70/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:10.815408 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:10.815425 247 storage/store.go:2593 [n1,s1,r107/2:/Table/70/1/5/{8-9}] removing replica r109/2 I190903 18:32:10.817371 63407 storage/replica_command.go:1430 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r106:/Table/70/1/5/{7-8} [(n3,s3):2, next=3, gen=63] I190903 18:32:10.825175 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:10.830641 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:10.851329 63407 storage/replica_raft.go:291 [n3,s3,r106/2:/Table/70/1/5/{7-8}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 18:32:10.857688 63407 storage/store_snapshot.go:995 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] sending LEARNER snapshot ffe1c7f0 at applied index 35 I190903 18:32:10.857895 63407 storage/store_snapshot.go:1038 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s E190903 18:32:10.858596 63543 storage/queue.go:1027 [n1,replicate,s1,r108/2:/Table/70/1/5{-/6}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:10.860086 63535 storage/replica_raftstorage.go:793 [n1,s1,r106/3:{-}] applying LEARNER snapshot [id=ffe1c7f0 index=35] I190903 18:32:10.864557 63535 storage/replica_raftstorage.go:814 [n1,s1,r106/3:/Table/70/1/5/{7-8}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=ffe1c7f0 index=35] I190903 18:32:10.867030 63407 storage/replica_command.go:1430 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] change replicas (add [(n1,s1):3] remove []): existing descriptor r106:/Table/70/1/5/{7-8} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=64] I190903 18:32:10.872089 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:10.874437 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:10.877043 63121 storage/replica_command.go:1430 [n1,replicate,s1,r82/3:/Table/{68/2-70}] change replicas (add [(n2,s2):4LEARNER] remove []): existing descriptor r82:/Table/{68/2-70} [(n1,s1):3, next=4, gen=57] I190903 18:32:10.878760 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:10.879646 63407 storage/replica_raft.go:291 [n3,s3,r106/2:/Table/70/1/5/{7-8}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 18:32:10.880265 63407 storage/replica_command.go:1430 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r106:/Table/70/1/5/{7-8} [(n3,s3):2, (n1,s1):3, next=4, gen=65] I190903 18:32:10.903799 63121 storage/replica_raft.go:291 [n1,s1,r82/3:/Table/{68/2-70}] proposing ADD_REPLICA[(n2,s2):4LEARNER]: after=[(n1,s1):3 (n2,s2):4LEARNER] next=5 I190903 18:32:10.905310 63121 storage/store_snapshot.go:995 [n1,replicate,s1,r82/3:/Table/{68/2-70}] sending LEARNER snapshot 82b55d01 at applied index 53 I190903 18:32:10.905493 63121 storage/store_snapshot.go:1038 [n1,replicate,s1,r82/3:/Table/{68/2-70}] streamed snapshot to (n2,s2):4LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:10.905540 63407 storage/replica_raft.go:291 [n3,s3,r106/2:/Table/70/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 18:32:10.905576 63407 storage/replica_raft.go:304 [n3,s3,r106/2:/Table/70/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:10.907165 63546 storage/replica_raftstorage.go:793 [n2,s2,r82/4:{-}] applying LEARNER snapshot [id=82b55d01 index=53] I190903 18:32:10.907853 63407 storage/replica_command.go:979 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r106 failed: storage/replica_raft.go:305: [n3,s3,r106/2:/Table/70/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:10.907882 63407 storage/queue.go:1027 [n3,replicate,s3,r106/2:/Table/70/1/5/{7-8}] change replicas of r106 failed: storage/replica_raft.go:305: [n3,s3,r106/2:/Table/70/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:10.941329 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:10.941906 63546 storage/replica_raftstorage.go:814 [n2,s2,r82/4:/Table/{68/2-70}] applied LEARNER snapshot [total=35ms ingestion=4@34ms id=82b55d01 index=53] I190903 18:32:10.942805 63121 storage/replica_command.go:1430 [n1,replicate,s1,r82/3:/Table/{68/2-70}] change replicas (add [(n2,s2):4] remove []): existing descriptor r82:/Table/{68/2-70} [(n1,s1):3, (n2,s2):4LEARNER, next=5, gen=58] I190903 18:32:10.946455 63121 storage/replica_raft.go:291 [n1,s1,r82/3:/Table/{68/2-70}] proposing ADD_REPLICA[(n2,s2):4]: after=[(n1,s1):3 (n2,s2):4] next=5 I190903 18:32:10.947006 63121 storage/replica_command.go:1430 [n1,replicate,s1,r82/3:/Table/{68/2-70}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r82:/Table/{68/2-70} [(n1,s1):3, (n2,s2):4, next=5, gen=59] I190903 18:32:10.964736 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:10.964833 63121 storage/replica_raft.go:291 [n1,s1,r82/3:/Table/{68/2-70}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 E190903 18:32:10.964878 63121 storage/replica_raft.go:304 [n1,s1,r82/3:/Table/{68/2-70}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 18:32:10.965692 63121 storage/replica_command.go:979 [n1,replicate,s1,r82/3:/Table/{68/2-70}] could not promote [] to voter, rolling back: change replicas of r82 failed: storage/replica_raft.go:305: [n1,s1,r82/3:/Table/{68/2-70}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) E190903 18:32:10.965727 63121 storage/queue.go:1027 [n1,replicate,s1,r82/3:/Table/{68/2-70}] change replicas of r82 failed: storage/replica_raft.go:305: [n1,s1,r82/3:/Table/{68/2-70}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 18:32:10.969274 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:10.987451 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:10.990319 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:10.994544 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:10.998095 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.000460 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.004647 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:11.006715 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.010827 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.013805 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.023503 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) E190903 18:32:11.024767 63986 storage/queue.go:1027 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] no removable replicas from range that needs a removal: [1*:11, 2:0] I190903 18:32:11.031455 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.044677 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.048716 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.055565 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:11.059128 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:11.080367 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:11.092385 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.101420 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.108096 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.114254 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.127480 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.129891 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.132963 64517 storage/replica_command.go:1430 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r103:/{Table/70/2-Max} [(n1,s1):2, next=3, gen=59] I190903 18:32:11.134792 64534 storage/replica_command.go:1430 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r113:/Table/70/1/5/{6-7} [(n2,s2):1, (n1,s1):2, next=3, gen=63] I190903 18:32:11.138840 64517 storage/replica_raft.go:291 [n1,s1,r103/2:/{Table/70/2-Max}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:11.142041 64517 storage/store_snapshot.go:995 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] sending LEARNER snapshot 77ec4ae4 at applied index 33 I190903 18:32:11.143154 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:11.143365 64517 storage/store_snapshot.go:1038 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:11.145208 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.146726 64444 storage/replica_raftstorage.go:793 [n2,s2,r103/3:{-}] applying LEARNER snapshot [id=77ec4ae4 index=33] I190903 18:32:11.152896 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.156854 64444 storage/replica_raftstorage.go:814 [n2,s2,r103/3:/{Table/70/2-Max}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=77ec4ae4 index=33] I190903 18:32:11.160903 64517 storage/replica_command.go:1430 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] change replicas (add [(n2,s2):3] remove []): existing descriptor r103:/{Table/70/2-Max} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=60] I190903 18:32:11.161465 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.167469 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:11.168568 64534 storage/replica_raft.go:291 [n2,s2,r113/1:/Table/70/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):1] next=3 I190903 18:32:11.170368 64534 storage/replica_command.go:1430 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r113:/Table/70/1/5/{6-7} [(n2,s2):1, next=3, gen=64] I190903 18:32:11.174700 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.177945 64517 storage/replica_raft.go:291 [n1,s1,r103/2:/{Table/70/2-Max}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:11.178604 64517 storage/replica_command.go:1430 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r103:/{Table/70/2-Max} [(n1,s1):2, (n2,s2):3, next=4, gen=61] I190903 18:32:11.192857 64517 storage/replica_raft.go:291 [n1,s1,r103/2:/{Table/70/2-Max}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:11.192955 64517 storage/replica_raft.go:304 [n1,s1,r103/2:/{Table/70/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:11.194239 64517 storage/replica_command.go:979 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] could not promote [] to voter, rolling back: change replicas of r103 failed: storage/replica_raft.go:305: [n1,s1,r103/2:/{Table/70/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:11.194345 64517 storage/queue.go:1027 [n1,replicate,s1,r103/2:/{Table/70/2-Max}] change replicas of r103 failed: storage/replica_raft.go:305: [n1,s1,r103/2:/{Table/70/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:11.194848 64747 storage/replica_command.go:1430 [n1,replicate,s1,r108/2:/Table/70/1/5{-/6}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r108:/Table/70/1/5{-/6} [(n2,s2):1, (n1,s1):2, next=3, gen=63] I190903 18:32:11.195349 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.206754 64534 storage/replica_raft.go:291 [n2,s2,r113/1:/Table/70/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):1 (n3,s3):3LEARNER] next=4 I190903 18:32:11.207566 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.208107 64747 storage/replica_raft.go:291 [n1,s1,r108/2:/Table/70/1/5{-/6}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:11.208196 64534 storage/store_snapshot.go:995 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] sending LEARNER snapshot a9ef1864 at applied index 17 I190903 18:32:11.208345 64534 storage/store_snapshot.go:1038 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:11.210029 64806 storage/replica_raftstorage.go:793 [n3,s3,r113/3:{-}] applying LEARNER snapshot [id=a9ef1864 index=17] I190903 18:32:11.218661 64588 storage/store.go:2593 [n1,replicaGC,s1,r113/2:/Table/70/1/5/{6-7}] removing replica r113/2 I190903 18:32:11.219255 64588 storage/replica_destroy.go:152 [n1,replicaGC,s1,r113/2:/Table/70/1/5/{6-7}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.219559 64826 storage/store.go:2593 [n2,replicaGC,s2,r108/1:/Table/70/1/5{-/6}] removing replica r108/1 I190903 18:32:11.219594 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.219872 64826 storage/replica_destroy.go:152 [n2,replicaGC,s2,r108/1:/Table/70/1/5{-/6}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.232923 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:11.233313 64702 storage/replica_command.go:1430 [n1,replicate,s1,r106/3:/Table/70/1/5/{7-8}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r106:/Table/70/1/5/{7-8} [(n3,s3):2, (n1,s1):3, next=4, gen=65] I190903 18:32:11.237756 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.243079 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.256518 64806 storage/replica_raftstorage.go:814 [n3,s3,r113/3:/Table/70/1/5/{6-7}] applied LEARNER snapshot [total=46ms ingestion=4@46ms id=a9ef1864 index=17] I190903 18:32:11.258948 64534 storage/replica_command.go:1430 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] change replicas (add [(n3,s3):3] remove []): existing descriptor r113:/Table/70/1/5/{6-7} [(n2,s2):1, (n3,s3):3LEARNER, next=4, gen=65] I190903 18:32:11.265059 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.270542 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.276994 64702 storage/replica_raft.go:291 [n1,s1,r106/3:/Table/70/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 18:32:11.279209 64534 storage/replica_raft.go:291 [n2,s2,r113/1:/Table/70/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):1 (n3,s3):3] next=4 I190903 18:32:11.280186 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.282410 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.282388 64534 storage/replica_command.go:1430 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r113:/Table/70/1/5/{6-7} [(n2,s2):1, (n3,s3):3, next=4, gen=66] I190903 18:32:11.286111 65093 storage/store.go:2593 [n3,replicaGC,s3,r106/2:/Table/70/1/5/{7-8}] removing replica r106/2 I190903 18:32:11.286360 65136 storage/replica_command.go:1430 [n1,replicate,s1,r101/2:/Table/70{-/1}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r101:/Table/70{-/1} [(n1,s1):2, next=3, gen=58] I190903 18:32:11.286597 65093 storage/replica_destroy.go:152 [n3,replicaGC,s3,r106/2:/Table/70/1/5/{7-8}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.297479 64534 storage/replica_raft.go:291 [n2,s2,r113/1:/Table/70/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 E190903 18:32:11.297556 64534 storage/replica_raft.go:304 [n2,s2,r113/1:/Table/70/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:11.298408 64534 storage/replica_command.go:979 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r113 failed: storage/replica_raft.go:305: [n2,s2,r113/1:/Table/70/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:11.298455 64534 storage/queue.go:1027 [n2,replicate,s2,r113/1:/Table/70/1/5/{6-7}] change replicas of r113 failed: storage/replica_raft.go:305: [n2,s2,r113/1:/Table/70/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:11.299781 65136 storage/replica_raft.go:291 [n1,s1,r101/2:/Table/70{-/1}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:11.301247 65136 storage/store_snapshot.go:995 [n1,replicate,s1,r101/2:/Table/70{-/1}] sending LEARNER snapshot c3ef8e73 at applied index 38 I190903 18:32:11.301404 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.301438 65136 storage/store_snapshot.go:1038 [n1,replicate,s1,r101/2:/Table/70{-/1}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:11.303167 63937 storage/replica_raftstorage.go:793 [n2,s2,r101/3:{-}] applying LEARNER snapshot [id=c3ef8e73 index=38] I190903 18:32:11.304437 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.316328 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.319323 63937 storage/replica_raftstorage.go:814 [n2,s2,r101/3:/Table/70{-/1}] applied LEARNER snapshot [total=15ms ingestion=4@12ms id=c3ef8e73 index=38] I190903 18:32:11.320222 65136 storage/replica_command.go:1430 [n1,replicate,s1,r101/2:/Table/70{-/1}] change replicas (add [(n2,s2):3] remove []): existing descriptor r101:/Table/70{-/1} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=59] I190903 18:32:11.336723 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 18:32:11.339255 65136 storage/replica_raft.go:291 [n1,s1,r101/2:/Table/70{-/1}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:11.339906 65136 storage/replica_command.go:1430 [n1,replicate,s1,r101/2:/Table/70{-/1}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r101:/Table/70{-/1} [(n1,s1):2, (n2,s2):3, next=4, gen=60] I190903 18:32:11.344691 65182 storage/replica_command.go:1430 [n2,replicate,s2,r82/4:/Table/{68/2-70}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r82:/Table/{68/2-70} [(n1,s1):3, (n2,s2):4, next=5, gen=59] I190903 18:32:11.348584 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:11.350672 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.357274 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.358116 65136 storage/replica_raft.go:291 [n1,s1,r101/2:/Table/70{-/1}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:11.358155 65136 storage/replica_raft.go:304 [n1,s1,r101/2:/Table/70{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:11.359116 65136 storage/replica_command.go:979 [n1,replicate,s1,r101/2:/Table/70{-/1}] could not promote [] to voter, rolling back: change replicas of r101 failed: storage/replica_raft.go:305: [n1,s1,r101/2:/Table/70{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:11.359159 65136 storage/queue.go:1027 [n1,replicate,s1,r101/2:/Table/70{-/1}] change replicas of r101 failed: storage/replica_raft.go:305: [n1,s1,r101/2:/Table/70{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:11.364655 65182 storage/replica_raft.go:291 [n2,s2,r82/4:/Table/{68/2-70}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 I190903 18:32:11.365105 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.373684 65267 storage/store.go:2593 [n1,replicaGC,s1,r82/3:/Table/{68/2-70}] removing replica r82/3 I190903 18:32:11.374123 65267 storage/replica_destroy.go:152 [n1,replicaGC,s1,r82/3:/Table/{68/2-70}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.381360 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 E190903 18:32:11.393640 65546 storage/queue.go:1027 [n3,replicate,s3,r113/3:/Table/70/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 3*:23] I190903 18:32:11.394965 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.402110 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.463653 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.466721 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.498176 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 18:32:11.503111 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 18:32:11.509721 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 18:32:11.521919 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 18:32:11.527055 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 18:32:11.529542 51129 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) === RUN TestRepartitioning/multi_col_list_partitioning_-_DEFAULT/multi_col_list_partitioning_-_DEFAULT_DEFAULT E190903 18:32:11.548529 65982 storage/queue.go:1027 [n2,replicate,s2,r101/3:/Table/70{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:45] I190903 18:32:11.558088 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=70 (multi col list partitioning) version=5 mtime=2019-09-03 18:32:11.532482077 +0000 UTC I190903 18:32:11.573133 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 69, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning"]} I190903 18:32:11.575399 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 4}] I190903 18:32:11.621549 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=70 (multi col list partitioning) version=6 mtime=2019-09-03 18:32:11.621376542 +0000 UTC I190903 18:32:11.685878 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 71, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} E190903 18:32:11.699832 67081 storage/queue.go:1027 [n2,replicate,s2,r103/3:/{Table/70/2-Max}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 18:32:11.700703 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 72, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:CREATE TABLE "multi col list partitioning - DEFAULT" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root} I190903 18:32:11.704294 67270 storage/replica_command.go:284 [n2,split,s2,r103/3:/{Table/70/2-Max}] initiating a split of this range at key /Table/72 [r114] (zone config) I190903 18:32:11.712620 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:11.714606 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:11.736806 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:11.762345 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:11.765825 67249 storage/replica_command.go:1430 [n2,replicate,s2,r101/3:/Table/70{-/1}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r101:/Table/70{-/1} [(n1,s1):2, (n2,s2):3, next=4, gen=60] I190903 18:32:11.775933 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:11.784318 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:11.785201 68404 storage/replica_command.go:284 [n2,split,s2,r114/2:/{Table/72-Max}] initiating a split of this range at key /Table/72/1 [r115] (zone config) I190903 18:32:11.786221 67249 storage/replica_raft.go:291 [n2,s2,r101/3:/Table/70{-/1}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:11.787413 66316 storage/replica_command.go:1430 [n2,replicate,s2,r103/3:/Table/7{0/2-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r103:/Table/7{0/2-2} [(n1,s1):2, (n2,s2):3, next=4, gen=62] I190903 18:32:11.807089 66316 storage/replica_raft.go:291 [n2,s2,r103/3:/Table/7{0/2-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:11.814649 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:11.814800 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:11.816717 68302 storage/replica_command.go:609 [n1,merge,s1,r107/2:/Table/70/1/{5/8-6}] initiating a merge of r110:/Table/70/{1/6-2} [(n1,s1):2, next=3, gen=65] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:11.823430 68636 storage/replica_command.go:284 [n2,split,s2,r115/2:/{Table/72/1-Max}] initiating a split of this range at key /Table/72/1/3/4 [r116] (zone config) I190903 18:32:11.824047 68921 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r114:/Table/72{-/1} [(n1,s1):1, (n2,s2):2, next=3, gen=63] I190903 18:32:11.826184 243 storage/store.go:2593 [n1,s1,r107/2:/Table/70/1/{5/8-6}] removing replica r110/2 I190903 18:32:11.838829 68433 storage/store.go:2593 [n1,replicaGC,s1,r101/2:/Table/70{-/1}] removing replica r101/2 I190903 18:32:11.839243 68433 storage/replica_destroy.go:152 [n1,replicaGC,s1,r101/2:/Table/70{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.839578 68667 storage/store.go:2593 [n1,replicaGC,s1,r103/2:/Table/7{0/2-2}] removing replica r103/2 I190903 18:32:11.839975 68667 storage/replica_destroy.go:152 [n1,replicaGC,s1,r103/2:/Table/7{0/2-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.840785 68921 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:11.863680 69069 storage/replica_command.go:284 [n2,split,s2,r116/2:/{Table/72/1/3…-Max}] initiating a split of this range at key /Table/72/1/3/5 [r117] (zone config) E190903 18:32:11.867919 68670 storage/queue.go:1027 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] no removable replicas from range that needs a removal: [1*:17, 2:0] I190903 18:32:11.889750 69042 storage/store.go:2593 [n1,replicaGC,s1,r114/1:/Table/72{-/1}] removing replica r114/1 I190903 18:32:11.890633 69042 storage/replica_destroy.go:152 [n1,replicaGC,s1,r114/1:/Table/72{-/1}] removed 5 (0+5) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:11.897165 69242 storage/replica_command.go:284 [n2,split,s2,r117/2:/{Table/72/1/3…-Max}] initiating a split of this range at key /Table/72/1/5 [r118] (zone config) I190903 18:32:11.898425 69070 storage/replica_command.go:1430 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r116:/Table/72/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=65] I190903 18:32:11.915680 69324 storage/replica_command.go:284 [n2,split,s2,r118/2:/{Table/72/1/5-Max}] initiating a split of this range at key /Table/72/1/5/7 [r119] (zone config) I190903 18:32:11.917010 69070 storage/replica_raft.go:291 [n2,s2,r116/2:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:11.917250 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:11.933181 69383 storage/store.go:2593 [n1,replicaGC,s1,r116/1:/Table/72/1/3/{4-5}] removing replica r116/1 I190903 18:32:11.933644 69383 storage/replica_destroy.go:152 [n1,replicaGC,s1,r116/1:/Table/72/1/3/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:11.934182 68947 storage/replica_command.go:1430 [n3,replicate,s3,r113/3:/Table/70/1/5/{6-7}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r113:/Table/70/1/5/{6-7} [(n2,s2):1, (n3,s3):3, next=4, gen=66] I190903 18:32:11.945133 69428 storage/replica_command.go:1430 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r115:/Table/72/1{-/3/4} [(n1,s1):1, (n2,s2):2, next=3, gen=64] I190903 18:32:12.012684 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.015222 69428 storage/replica_raft.go:291 [n1,s1,r115/1:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 I190903 18:32:12.020074 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:12.024725 68947 storage/replica_raft.go:291 [n3,s3,r113/3:/Table/70/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 I190903 18:32:12.025419 69577 storage/replica_command.go:1430 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r117:/Table/72/1/{3/5-5} [(n1,s1):1, (n2,s2):2, next=3, gen=66] I190903 18:32:12.026974 69539 storage/store.go:2593 [n2,replicaGC,s2,r115/2:/Table/72/1{-/3/4}] removing replica r115/2 I190903 18:32:12.026993 69482 storage/replica_command.go:1430 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r118:/Table/72/1/5{-/7} [(n1,s1):1, (n2,s2):2, next=3, gen=67] I190903 18:32:12.027009 69483 storage/replica_command.go:284 [n2,split,s2,r119/2:/{Table/72/1/5…-Max}] initiating a split of this range at key /Table/72/1/5/8 [r120] (zone config) I190903 18:32:12.027456 69539 storage/replica_destroy.go:152 [n2,replicaGC,s2,r115/2:/Table/72/1{-/3/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:12.029244 69577 storage/replica_raft.go:291 [n1,s1,r117/1:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 I190903 18:32:12.037005 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:12.040063 69704 storage/store.go:2593 [n2,replicaGC,s2,r113/1:/Table/70/1/5/{6-7}] removing replica r113/1 I190903 18:32:12.040440 69704 storage/replica_destroy.go:152 [n2,replicaGC,s2,r113/1:/Table/70/1/5/{6-7}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:12.050298 69482 storage/replica_raft.go:291 [n2,s2,r118/2:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:12.063941 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:12.065530 69776 storage/replica_command.go:284 [n2,split,s2,r120/2:/{Table/72/1/5…-Max}] initiating a split of this range at key /Table/72/1/5/9 [r121] (zone config) I190903 18:32:12.066338 69734 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r119:/Table/72/1/5/{7-8} [(n1,s1):1, (n2,s2):2, next=3, gen=68] I190903 18:32:12.068594 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:12.074004 69887 storage/store.go:2593 [n1,replicaGC,s1,r118/1:/Table/72/1/5{-/7}] removing replica r118/1 I190903 18:32:12.077450 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:12.078651 69734 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/72/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:12.079022 69887 storage/replica_destroy.go:152 [n1,replicaGC,s1,r118/1:/Table/72/1/5{-/7}] removed 5 (0+5) keys in 5ms [clear=5ms commit=0ms] I190903 18:32:12.079725 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.084481 69740 storage/store.go:2593 [n2,replicaGC,s2,r117/2:/Table/72/1/{3/5-5}] removing replica r117/2 I190903 18:32:12.084747 69740 storage/replica_destroy.go:152 [n2,replicaGC,s2,r117/2:/Table/72/1/{3/5-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:12.088556 69734 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r119:/Table/72/1/5/{7-8} [(n2,s2):2, next=3, gen=69] I190903 18:32:12.091660 69742 storage/store.go:2593 [n1,replicaGC,s1,r119/1:/Table/72/1/5/{7-8}] removing replica r119/1 I190903 18:32:12.094191 69742 storage/replica_destroy.go:152 [n1,replicaGC,s1,r119/1:/Table/72/1/5/{7-8}] removed 5 (0+5) keys in 2ms [clear=2ms commit=0ms] I190903 18:32:12.095752 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.095869 69712 storage/replica_command.go:284 [n2,split,s2,r121/2:/{Table/72/1/5…-Max}] initiating a split of this range at key /Table/72/1/6 [r122] (zone config) I190903 18:32:12.100417 69734 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/72/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:12.101069 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:12.117489 69734 storage/store_snapshot.go:995 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] sending LEARNER snapshot c29b1690 at applied index 19 I190903 18:32:12.122840 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.124626 69734 storage/store_snapshot.go:1038 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:12.143656 69746 storage/replica_raftstorage.go:793 [n3,s3,r119/3:{-}] applying LEARNER snapshot [id=c29b1690 index=19] I190903 18:32:12.145464 69746 storage/replica_raftstorage.go:814 [n3,s3,r119/3:/Table/72/1/5/{7-8}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=c29b1690 index=19] I190903 18:32:12.147513 69734 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] change replicas (add [(n3,s3):3] remove []): existing descriptor r119:/Table/72/1/5/{7-8} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=70] I190903 18:32:12.152731 70095 storage/replica_command.go:284 [n2,split,s2,r122/2:/{Table/72/1/6-Max}] initiating a split of this range at key /Table/72/2 [r123] (zone config) I190903 18:32:12.160675 69734 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/72/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:12.161586 69734 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r119:/Table/72/1/5/{7-8} [(n2,s2):2, (n3,s3):3, next=4, gen=71] I190903 18:32:12.162197 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.169799 69734 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/72/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:12.169847 69734 storage/replica_raft.go:304 [n2,s2,r119/2:/Table/72/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:12.170410 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:12.172527 69734 storage/replica_command.go:979 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r119 failed: storage/replica_raft.go:305: [n2,s2,r119/2:/Table/72/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:12.172609 69734 storage/queue.go:1027 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] change replicas of r119 failed: storage/replica_raft.go:305: [n2,s2,r119/2:/Table/72/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:12.174898 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:12.175009 70365 storage/replica_command.go:1430 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r121:/Table/72/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=70] E190903 18:32:12.177118 70330 storage/queue.go:1027 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] no removable replicas from range that needs a removal: [1*:17, 2:0] I190903 18:32:12.198483 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:12.204965 70365 storage/replica_raft.go:291 [n2,s2,r121/2:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:12.210935 70495 storage/queue.go:1027 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] no removable replicas from range that needs a removal: [1*:18, 2:0] E190903 18:32:12.210945 70596 storage/queue.go:1027 [n3,replicate,s3,r119/3:/Table/72/1/5/{7-8}] no removable replicas from range that needs a removal: [2:0, 3*:27] I190903 18:32:12.215924 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:12.218770 70580 storage/store.go:2593 [n1,replicaGC,s1,r121/1:/Table/72/1/{5/9-6}] removing replica r121/1 I190903 18:32:12.219354 70580 storage/replica_destroy.go:152 [n1,replicaGC,s1,r121/1:/Table/72/1/{5/9-6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:12.222769 70661 storage/queue.go:1027 [n2,replicate,s2,r123/2:/{Table/72/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:11] I190903 18:32:12.223880 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.255859 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.308316 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.318658 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.322966 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:12.327538 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:12.329395 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:12.331562 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:12.336247 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:12.339576 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.346010 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=2 mtime=2019-09-03 18:32:12.343195816 +0000 UTC I190903 18:32:12.358052 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 1}] I190903 18:32:12.362382 71025 sql/lease.go:923 new lease: 72("multi col list partitioning - DEFAULT DEFAULT") ver=2:1567535843.779586670,0, refcount=0 I190903 18:32:12.404654 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=3 mtime=2019-09-03 18:32:12.404477839 +0000 UTC I190903 18:32:12.410078 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 2}] I190903 18:32:12.412625 71455 sql/lease.go:923 new lease: 72("multi col list partitioning - DEFAULT DEFAULT") ver=3:1567535843.780586670,0, refcount=0 I190903 18:32:12.437205 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=4 mtime=2019-09-03 18:32:12.436458923 +0000 UTC I190903 18:32:12.444861 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 72, info: {TableName:data.public."multi col list partitioning - DEFAULT DEFAULT" Statement:ALTER TABLE "multi col list partitioning - DEFAULT DEFAULT" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT)), PARTITION pd VALUES IN ((DEFAULT, DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:12.446993 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 3}] I190903 18:32:12.461715 71808 sql/lease.go:923 new lease: 72("multi col list partitioning - DEFAULT DEFAULT") ver=4:1567535843.781586670,0, refcount=0 I190903 18:32:12.586218 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:12.600895 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:12.610632 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:12.620669 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:12.638018 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION pd OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:12.640702 73480 storage/replica_command.go:1430 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r122:/Table/72/{1/6-2} [(n1,s1):1, (n2,s2):2, next=3, gen=71] I190903 18:32:12.644514 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:12.647573 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:12.651709 73480 storage/replica_raft.go:291 [n1,s1,r122/1:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 E190903 18:32:12.654892 73559 storage/queue.go:1027 [n2,replicate,s2,r120/2:/Table/72/1/5/{8-9}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 18:32:12.655343 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.660925 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.666186 73576 storage/store.go:2593 [n2,replicaGC,s2,r122/2:/Table/72/{1/6-2}] removing replica r122/2 I190903 18:32:12.666578 73576 storage/replica_destroy.go:152 [n2,replicaGC,s2,r122/2:/Table/72/{1/6-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:12.692233 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.697400 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 E190903 18:32:12.700140 73811 storage/queue.go:1027 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] no removable replicas from range that needs a removal: [2*:30, 3:0] I190903 18:32:12.704294 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.711224 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.720079 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.723835 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.733798 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.745607 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:12.749349 74137 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/72/1/5/{7-8}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r119:/Table/72/1/5/{7-8} [(n2,s2):2, (n3,s3):3, next=4, gen=71] I190903 18:32:12.750869 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.762444 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.766493 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:12.775811 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.775969 74137 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/72/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):2] next=4 I190903 18:32:12.777237 74221 storage/replica_command.go:1430 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r118:/Table/72/1/5{-/7} [(n2,s2):2, next=3, gen=68] I190903 18:32:12.783823 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:12.801372 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:12.801818 74206 storage/store.go:2593 [n3,replicaGC,s3,r119/3:/Table/72/1/5/{7-8}] removing replica r119/3 I190903 18:32:12.802306 74206 storage/replica_destroy.go:152 [n3,replicaGC,s3,r119/3:/Table/72/1/5/{7-8}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:12.805861 74221 storage/replica_raft.go:291 [n2,s2,r118/2:/Table/72/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:12.807735 74221 storage/store_snapshot.go:995 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] sending LEARNER snapshot fb55506e at applied index 22 I190903 18:32:12.808118 74221 storage/store_snapshot.go:1038 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:12.808662 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:12.812778 74077 storage/replica_raftstorage.go:793 [n1,s1,r118/3:{-}] applying LEARNER snapshot [id=fb55506e index=22] I190903 18:32:12.817703 73070 storage/replica_command.go:609 [n1,merge,s1,r105/2:/Table/70/1/{3/5-5}] initiating a merge of r108:/Table/70/1/5{-/6} [(n1,s1):2, next=3, gen=64] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:12.817798 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:12.822553 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:12.825326 74077 storage/replica_raftstorage.go:814 [n1,s1,r118/3:/Table/72/1/5{-/7}] applied LEARNER snapshot [total=12ms ingestion=4@11ms id=fb55506e index=22] I190903 18:32:12.825826 74221 storage/replica_command.go:1430 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r118:/Table/72/1/5{-/7} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=69] I190903 18:32:12.826692 223 storage/store.go:2593 [n1,s1,r105/2:/Table/70/1/{3/5-5}] removing replica r108/2 I190903 18:32:12.827660 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:12.832659 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:12.847775 74221 storage/replica_raft.go:291 [n2,s2,r118/2:/Table/72/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:12.848610 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:12.849619 74221 storage/replica_command.go:1430 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r118:/Table/72/1/5{-/7} [(n2,s2):2, (n1,s1):3, next=4, gen=70] I190903 18:32:12.865211 74221 storage/replica_raft.go:291 [n2,s2,r118/2:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:12.865262 74221 storage/replica_raft.go:304 [n2,s2,r118/2:/Table/72/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:12.867528 74221 storage/replica_command.go:979 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r118 failed: storage/replica_raft.go:305: [n2,s2,r118/2:/Table/72/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:12.867574 74221 storage/queue.go:1027 [n2,replicate,s2,r118/2:/Table/72/1/5{-/7}] change replicas of r118 failed: storage/replica_raft.go:305: [n2,s2,r118/2:/Table/72/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:12.868057 74578 storage/replica_command.go:1430 [n2,replicate,s2,r35/2:/Table/68{-/1}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r35:/Table/68{-/1} [(n2,s2):2, next=3, gen=47] I190903 18:32:12.877442 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:12.885681 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:12.897009 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:12.899057 74578 storage/replica_raft.go:291 [n2,s2,r35/2:/Table/68{-/1}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:12.900742 74578 storage/store_snapshot.go:995 [n2,replicate,s2,r35/2:/Table/68{-/1}] sending LEARNER snapshot c40075ac at applied index 37 I190903 18:32:12.900907 74578 storage/store_snapshot.go:1038 [n2,replicate,s2,r35/2:/Table/68{-/1}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:12.908138 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:12.929824 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:12.952378 74712 storage/replica_raftstorage.go:793 [n3,s3,r35/3:{-}] applying LEARNER snapshot [id=c40075ac index=37] I190903 18:32:12.954318 74712 storage/replica_raftstorage.go:814 [n3,s3,r35/3:/Table/68{-/1}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=c40075ac index=37] I190903 18:32:12.955313 74578 storage/replica_command.go:1430 [n2,replicate,s2,r35/2:/Table/68{-/1}] change replicas (add [(n3,s3):3] remove []): existing descriptor r35:/Table/68{-/1} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=48] I190903 18:32:12.971332 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:13.015660 74578 storage/replica_raft.go:291 [n2,s2,r35/2:/Table/68{-/1}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:13.016275 74578 storage/replica_command.go:1430 [n2,replicate,s2,r35/2:/Table/68{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r35:/Table/68{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=49] I190903 18:32:13.064077 74578 storage/replica_raft.go:291 [n2,s2,r35/2:/Table/68{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:13.064126 74578 storage/replica_raft.go:304 [n2,s2,r35/2:/Table/68{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:13.066462 74578 storage/replica_command.go:979 [n2,replicate,s2,r35/2:/Table/68{-/1}] could not promote [] to voter, rolling back: change replicas of r35 failed: storage/replica_raft.go:305: [n2,s2,r35/2:/Table/68{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:13.066495 74578 storage/queue.go:1027 [n2,replicate,s2,r35/2:/Table/68{-/1}] change replicas of r35 failed: storage/replica_raft.go:305: [n2,s2,r35/2:/Table/68{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:13.077187 75198 storage/queue.go:1027 [n1,replicate,s1,r123/1:/{Table/72/2-Max}] no removable replicas from range that needs a removal: [1*:17, 2:0] I190903 18:32:13.077976 75217 storage/replica_command.go:1430 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r121:/Table/72/1/{5/9-6} [(n2,s2):2, next=3, gen=71] I190903 18:32:13.078875 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) E190903 18:32:13.079028 75051 storage/queue.go:1027 [n3,replicate,s3,r35/3:/Table/68{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:44] I190903 18:32:13.082852 75259 storage/replica_command.go:1430 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r115:/Table/72/1{-/3/4} [(n1,s1):1, next=3, gen=65] I190903 18:32:13.096323 75217 storage/replica_raft.go:291 [n2,s2,r121/2:/Table/72/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:13.097147 75259 storage/replica_raft.go:291 [n1,s1,r115/1:/Table/72/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n2,s2):3LEARNER] next=4 I190903 18:32:13.100137 75217 storage/store_snapshot.go:995 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] sending LEARNER snapshot 3d0b0155 at applied index 23 I190903 18:32:13.101536 75217 storage/store_snapshot.go:1038 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:13.104606 75259 storage/store_snapshot.go:995 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] sending LEARNER snapshot b19c2a12 at applied index 25 I190903 18:32:13.104745 75259 storage/store_snapshot.go:1038 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:13.109631 75167 storage/replica_raftstorage.go:793 [n2,s2,r115/3:{-}] applying LEARNER snapshot [id=b19c2a12 index=25] I190903 18:32:13.120432 75308 storage/replica_raftstorage.go:793 [n1,s1,r121/3:{-}] applying LEARNER snapshot [id=3d0b0155 index=23] I190903 18:32:13.121755 75308 storage/replica_raftstorage.go:814 [n1,s1,r121/3:/Table/72/1/{5/9-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=3d0b0155 index=23] I190903 18:32:13.123072 75217 storage/replica_command.go:1430 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] change replicas (add [(n1,s1):3] remove []): existing descriptor r121:/Table/72/1/{5/9-6} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=72] I190903 18:32:13.132780 75217 storage/replica_raft.go:291 [n2,s2,r121/2:/Table/72/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:13.133531 75217 storage/replica_command.go:1430 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r121:/Table/72/1/{5/9-6} [(n2,s2):2, (n1,s1):3, next=4, gen=73] I190903 18:32:13.147269 75167 storage/replica_raftstorage.go:814 [n2,s2,r115/3:/Table/72/1{-/3/4}] applied LEARNER snapshot [total=38ms ingestion=4@37ms id=b19c2a12 index=25] I190903 18:32:13.154936 75259 storage/replica_command.go:1430 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] change replicas (add [(n2,s2):3] remove []): existing descriptor r115:/Table/72/1{-/3/4} [(n1,s1):1, (n2,s2):3LEARNER, next=4, gen=66] I190903 18:32:13.158753 75259 storage/replica_raft.go:291 [n1,s1,r115/1:/Table/72/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n2,s2):3] next=4 I190903 18:32:13.161954 75259 storage/replica_command.go:1430 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r115:/Table/72/1{-/3/4} [(n1,s1):1, (n2,s2):3, next=4, gen=67] I190903 18:32:13.173664 75217 storage/replica_raft.go:291 [n2,s2,r121/2:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:13.173705 75217 storage/replica_raft.go:304 [n2,s2,r121/2:/Table/72/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:13.174528 75217 storage/replica_command.go:979 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r121 failed: storage/replica_raft.go:305: [n2,s2,r121/2:/Table/72/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:13.174556 75217 storage/queue.go:1027 [n2,replicate,s2,r121/2:/Table/72/1/{5/9-6}] change replicas of r121 failed: storage/replica_raft.go:305: [n2,s2,r121/2:/Table/72/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:13.174979 75530 storage/replica_command.go:1430 [n2,replicate,s2,r41/2:/Table/58{-/1}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r41:/Table/58{-/1} [(n2,s2):2, next=3, gen=16] I190903 18:32:13.183561 75259 storage/replica_raft.go:291 [n1,s1,r115/1:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 E190903 18:32:13.183594 75259 storage/replica_raft.go:304 [n1,s1,r115/1:/Table/72/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:13.184555 75259 storage/replica_command.go:979 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r115 failed: storage/replica_raft.go:305: [n1,s1,r115/1:/Table/72/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:13.184586 75259 storage/queue.go:1027 [n1,replicate,s1,r115/1:/Table/72/1{-/3/4}] change replicas of r115 failed: storage/replica_raft.go:305: [n1,s1,r115/1:/Table/72/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:13.185917 75575 storage/replica_command.go:1430 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r120:/Table/72/1/5/{8-9} [(n1,s1):1, (n2,s2):2, next=3, gen=69] I190903 18:32:13.196290 75575 storage/replica_raft.go:291 [n1,s1,r120/1:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 I190903 18:32:13.203953 75575 storage/replica_command.go:1430 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r120:/Table/72/1/5/{8-9} [(n1,s1):1, next=3, gen=70] I190903 18:32:13.213169 75037 storage/store.go:2593 [n2,replicaGC,s2,r120/2:/Table/72/1/5/{8-9}] removing replica r120/2 I190903 18:32:13.213178 75530 storage/replica_raft.go:291 [n2,s2,r41/2:/Table/58{-/1}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:13.214418 75575 storage/replica_raft.go:291 [n1,s1,r120/1:/Table/72/1/5/{8-9}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n3,s3):3LEARNER] next=4 I190903 18:32:13.214233 75037 storage/replica_destroy.go:152 [n2,replicaGC,s2,r120/2:/Table/72/1/5/{8-9}] removed 5 (0+5) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:13.216448 75575 storage/store_snapshot.go:995 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] sending LEARNER snapshot 5a93c7e5 at applied index 31 I190903 18:32:13.216451 75530 storage/store_snapshot.go:995 [n2,replicate,s2,r41/2:/Table/58{-/1}] sending LEARNER snapshot b1b12cee at applied index 40 I190903 18:32:13.216602 75575 storage/store_snapshot.go:1038 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:13.216664 75530 storage/store_snapshot.go:1038 [n2,replicate,s2,r41/2:/Table/58{-/1}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:13.217334 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:13.218116 75624 storage/replica_raftstorage.go:793 [n3,s3,r41/3:{-}] applying LEARNER snapshot [id=b1b12cee index=40] I190903 18:32:13.219577 75624 storage/replica_raftstorage.go:814 [n3,s3,r41/3:/Table/58{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=b1b12cee index=40] I190903 18:32:13.221589 75530 storage/replica_command.go:1430 [n2,replicate,s2,r41/2:/Table/58{-/1}] change replicas (add [(n3,s3):3] remove []): existing descriptor r41:/Table/58{-/1} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=17] I190903 18:32:13.228864 75483 storage/replica_raftstorage.go:793 [n3,s3,r120/3:{-}] applying LEARNER snapshot [id=5a93c7e5 index=31] I190903 18:32:13.264777 75483 storage/replica_raftstorage.go:814 [n3,s3,r120/3:/Table/72/1/5/{8-9}] applied LEARNER snapshot [total=36ms ingestion=4@35ms id=5a93c7e5 index=31] I190903 18:32:13.265346 75575 storage/replica_command.go:1430 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] change replicas (add [(n3,s3):3] remove []): existing descriptor r120:/Table/72/1/5/{8-9} [(n1,s1):1, (n3,s3):3LEARNER, next=4, gen=71] I190903 18:32:13.265353 75530 storage/replica_raft.go:291 [n2,s2,r41/2:/Table/58{-/1}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:13.266043 75530 storage/replica_command.go:1430 [n2,replicate,s2,r41/2:/Table/58{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r41:/Table/58{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=18] I190903 18:32:13.280621 75575 storage/replica_raft.go:291 [n1,s1,r120/1:/Table/72/1/5/{8-9}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 18:32:13.281143 75575 storage/replica_command.go:1430 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r120:/Table/72/1/5/{8-9} [(n1,s1):1, (n3,s3):3, next=4, gen=72] I190903 18:32:13.287290 75530 storage/replica_raft.go:291 [n2,s2,r41/2:/Table/58{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:13.287326 75530 storage/replica_raft.go:304 [n2,s2,r41/2:/Table/58{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:13.289054 75530 storage/replica_command.go:979 [n2,replicate,s2,r41/2:/Table/58{-/1}] could not promote [] to voter, rolling back: change replicas of r41 failed: storage/replica_raft.go:305: [n2,s2,r41/2:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:13.290198 75530 storage/queue.go:1027 [n2,replicate,s2,r41/2:/Table/58{-/1}] change replicas of r41 failed: storage/replica_raft.go:305: [n2,s2,r41/2:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:13.290755 75905 storage/replica_command.go:1430 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r116:/Table/72/1/3/{4-5} [(n2,s2):2, next=3, gen=66] I190903 18:32:13.292277 75575 storage/replica_raft.go:291 [n1,s1,r120/1:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 E190903 18:32:13.292323 75575 storage/replica_raft.go:304 [n1,s1,r120/1:/Table/72/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:13.295634 75575 storage/replica_command.go:979 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r120 failed: storage/replica_raft.go:305: [n1,s1,r120/1:/Table/72/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:13.295679 75575 storage/queue.go:1027 [n1,replicate,s1,r120/1:/Table/72/1/5/{8-9}] change replicas of r120 failed: storage/replica_raft.go:305: [n1,s1,r120/1:/Table/72/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:13.297941 75916 storage/replica_command.go:1430 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r122:/Table/72/{1/6-2} [(n1,s1):1, next=3, gen=72] E190903 18:32:13.306990 75784 storage/queue.go:1027 [n3,replicate,s3,r120/3:/Table/72/1/5/{8-9}] no removable replicas from range that needs a removal: [1:0, 3*:37] I190903 18:32:13.317697 75916 storage/replica_raft.go:291 [n1,s1,r122/1:/Table/72/{1/6-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n2,s2):3LEARNER] next=4 I190903 18:32:13.318790 75916 storage/store_snapshot.go:995 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] sending LEARNER snapshot 4b5e817d at applied index 25 I190903 18:32:13.319370 75916 storage/store_snapshot.go:1038 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:13.320624 75921 storage/replica_raftstorage.go:793 [n2,s2,r122/3:{-}] applying LEARNER snapshot [id=4b5e817d index=25] I190903 18:32:13.323185 75921 storage/replica_raftstorage.go:814 [n2,s2,r122/3:/Table/72/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=4b5e817d index=25] I190903 18:32:13.329746 75916 storage/replica_command.go:1430 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r122:/Table/72/{1/6-2} [(n1,s1):1, (n2,s2):3LEARNER, next=4, gen=73] I190903 18:32:13.333317 75916 storage/replica_raft.go:291 [n1,s1,r122/1:/Table/72/{1/6-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n2,s2):3] next=4 I190903 18:32:13.333827 75916 storage/replica_command.go:1430 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r122:/Table/72/{1/6-2} [(n1,s1):1, (n2,s2):3, next=4, gen=74] I190903 18:32:13.334521 75905 storage/replica_raft.go:291 [n2,s2,r116/2:/Table/72/1/3/{4-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:13.335704 75905 storage/store_snapshot.go:995 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] sending LEARNER snapshot 7055a9ea at applied index 25 I190903 18:32:13.335933 75905 storage/store_snapshot.go:1038 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:13.337044 75838 storage/replica_raftstorage.go:793 [n1,s1,r116/3:{-}] applying LEARNER snapshot [id=7055a9ea index=25] I190903 18:32:13.358042 75838 storage/replica_raftstorage.go:814 [n1,s1,r116/3:/Table/72/1/3/{4-5}] applied LEARNER snapshot [total=21ms ingestion=4@20ms id=7055a9ea index=25] I190903 18:32:13.358914 76137 storage/replica_command.go:1430 [n3,replicate,s3,r120/3:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r120:/Table/72/1/5/{8-9} [(n1,s1):1, (n3,s3):3, next=4, gen=72] I190903 18:32:13.359438 75905 storage/replica_command.go:1430 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r116:/Table/72/1/3/{4-5} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=67] I190903 18:32:13.365691 75916 storage/replica_raft.go:291 [n1,s1,r122/1:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 E190903 18:32:13.365728 75916 storage/replica_raft.go:304 [n1,s1,r122/1:/Table/72/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:13.370630 75905 storage/replica_raft.go:291 [n2,s2,r116/2:/Table/72/1/3/{4-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:13.371192 75905 storage/replica_command.go:1430 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r116:/Table/72/1/3/{4-5} [(n2,s2):2, (n1,s1):3, next=4, gen=68] I190903 18:32:13.372157 75916 storage/replica_command.go:979 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r122 failed: storage/replica_raft.go:305: [n1,s1,r122/1:/Table/72/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:13.372656 75916 storage/queue.go:1027 [n1,replicate,s1,r122/1:/Table/72/{1/6-2}] change replicas of r122 failed: storage/replica_raft.go:305: [n1,s1,r122/1:/Table/72/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:13.373056 75331 storage/replica_command.go:1430 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r117:/Table/72/1/{3/5-5} [(n1,s1):1, next=3, gen=67] I190903 18:32:13.389496 75331 storage/replica_raft.go:291 [n1,s1,r117/1:/Table/72/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n2,s2):3LEARNER] next=4 I190903 18:32:13.394932 75331 storage/store_snapshot.go:995 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] sending LEARNER snapshot d24b1dfa at applied index 27 I190903 18:32:13.395100 75331 storage/store_snapshot.go:1038 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:13.396811 76203 storage/replica_raftstorage.go:793 [n2,s2,r117/3:{-}] applying LEARNER snapshot [id=d24b1dfa index=27] I190903 18:32:13.398117 76203 storage/replica_raftstorage.go:814 [n2,s2,r117/3:/Table/72/1/{3/5-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=d24b1dfa index=27] I190903 18:32:13.398907 75331 storage/replica_command.go:1430 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r117:/Table/72/1/{3/5-5} [(n1,s1):1, (n2,s2):3LEARNER, next=4, gen=68] I190903 18:32:13.420738 76137 storage/replica_raft.go:291 [n3,s3,r120/3:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 I190903 18:32:13.423435 75331 storage/replica_raft.go:291 [n1,s1,r117/1:/Table/72/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n2,s2):3] next=4 I190903 18:32:13.423760 75905 storage/replica_raft.go:291 [n2,s2,r116/2:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:13.423809 75905 storage/replica_raft.go:304 [n2,s2,r116/2:/Table/72/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:13.423953 75331 storage/replica_command.go:1430 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r117:/Table/72/1/{3/5-5} [(n1,s1):1, (n2,s2):3, next=4, gen=69] I190903 18:32:13.425442 75905 storage/replica_command.go:979 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r116 failed: storage/replica_raft.go:305: [n2,s2,r116/2:/Table/72/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:13.425483 75905 storage/queue.go:1027 [n2,replicate,s2,r116/2:/Table/72/1/3/{4-5}] change replicas of r116 failed: storage/replica_raft.go:305: [n2,s2,r116/2:/Table/72/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:13.429291 76188 storage/queue.go:1027 [n3,replicate,s3,r41/3:/Table/58{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:47] I190903 18:32:13.439513 75331 storage/replica_raft.go:291 [n1,s1,r117/1:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 E190903 18:32:13.439559 75331 storage/replica_raft.go:304 [n1,s1,r117/1:/Table/72/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:13.440540 75331 storage/replica_command.go:979 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r117 failed: storage/replica_raft.go:305: [n1,s1,r117/1:/Table/72/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:13.440569 75331 storage/queue.go:1027 [n1,replicate,s1,r117/1:/Table/72/1/{3/5-5}] change replicas of r117 failed: storage/replica_raft.go:305: [n1,s1,r117/1:/Table/72/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:13.453763 76425 storage/queue.go:1027 [n1,replicate,s1,r116/3:/Table/72/1/3/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:32] I190903 18:32:13.470735 76524 storage/replica_command.go:1430 [n1,replicate,s1,r123/1:/{Table/72/2-Max}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r123:/{Table/72/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=71] I190903 18:32:13.492731 76362 storage/store.go:2593 [n1,replicaGC,s1,r120/1:/Table/72/1/5/{8-9}] removing replica r120/1 I190903 18:32:13.493177 76362 storage/replica_destroy.go:152 [n1,replicaGC,s1,r120/1:/Table/72/1/5/{8-9}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:13.495626 76524 storage/replica_raft.go:291 [n1,s1,r123/1:/{Table/72/2-Max}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 I190903 18:32:13.508903 76616 storage/store.go:2593 [n2,replicaGC,s2,r123/2:/{Table/72/2-Max}] removing replica r123/2 I190903 18:32:13.509338 76616 storage/replica_destroy.go:152 [n2,replicaGC,s2,r123/2:/{Table/72/2-Max}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:13.533639 76017 storage/replica_command.go:1430 [n3,replicate,s3,r35/3:/Table/68{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r35:/Table/68{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=49] I190903 18:32:13.538046 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:13.545105 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:13.551848 76628 storage/replica_command.go:1430 [n1,replicate,s1,r118/3:/Table/72/1/5{-/7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r118:/Table/72/1/5{-/7} [(n2,s2):2, (n1,s1):3, next=4, gen=70] I190903 18:32:13.555605 76628 storage/replica_raft.go:291 [n1,s1,r118/3:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:13.585579 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:13.591014 76738 storage/store.go:2593 [n2,replicaGC,s2,r118/2:/Table/72/1/5{-/7}] removing replica r118/2 I190903 18:32:13.591361 76738 storage/replica_destroy.go:152 [n2,replicaGC,s2,r118/2:/Table/72/1/5{-/7}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:13.591796 76017 storage/replica_raft.go:291 [n3,s3,r35/3:/Table/68{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:13.591879 76827 storage/queue.go:1027 [n2,replicate,s2,r115/3:/Table/72/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 3*:33] I190903 18:32:13.644984 76782 storage/store.go:2593 [n2,replicaGC,s2,r35/2:/Table/68{-/1}] removing replica r35/2 I190903 18:32:13.645417 76782 storage/replica_destroy.go:152 [n2,replicaGC,s2,r35/2:/Table/68{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:13.657264 77141 storage/queue.go:1027 [n2,replicate,s2,r117/3:/Table/72/1/{3/5-5}] no removable replicas from range that needs a removal: [1:0, 3*:34] I190903 18:32:13.732311 77151 storage/replica_command.go:1430 [n3,replicate,s3,r41/3:/Table/58{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r41:/Table/58{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=18] E190903 18:32:13.740469 77343 storage/queue.go:1027 [n2,replicate,s2,r122/3:/Table/72/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 3*:33] I190903 18:32:13.741384 77151 storage/replica_raft.go:291 [n3,s3,r41/3:/Table/58{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:13.749630 77355 storage/store.go:2593 [n2,replicaGC,s2,r41/2:/Table/58{-/1}] removing replica r41/2 I190903 18:32:13.750037 77355 storage/replica_destroy.go:152 [n2,replicaGC,s2,r41/2:/Table/58{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:13.786582 77509 storage/replica_command.go:1430 [n2,replicate,s2,r115/3:/Table/72/1{-/3/4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r115:/Table/72/1{-/3/4} [(n1,s1):1, (n2,s2):3, next=4, gen=67] I190903 18:32:13.793274 77509 storage/replica_raft.go:291 [n2,s2,r115/3:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 I190903 18:32:13.853555 77541 storage/store.go:2593 [n1,replicaGC,s1,r115/1:/Table/72/1{-/3/4}] removing replica r115/1 I190903 18:32:13.853921 77541 storage/replica_destroy.go:152 [n1,replicaGC,s1,r115/1:/Table/72/1{-/3/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:13.882283 77658 storage/replica_command.go:1430 [n2,replicate,s2,r122/3:/Table/72/{1/6-2}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r122:/Table/72/{1/6-2} [(n1,s1):1, (n2,s2):3, next=4, gen=74] I190903 18:32:13.889624 77658 storage/replica_raft.go:291 [n2,s2,r122/3:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 E190903 18:32:13.894215 77775 storage/queue.go:1027 [n1,replicate,s1,r121/3:/Table/72/1/{5/9-6}] no removable replicas from range that needs a removal: [2:0, 3*:31] I190903 18:32:13.899142 276 storage/compactor/compactor.go:370 [n1,s1,compactor] processing compaction #1-39/39 (/System/tsd-/Max) for 93 KiB (reasons: size=false used=true avail=false) I190903 18:32:13.899214 77838 storage/store.go:2593 [n1,replicaGC,s1,r122/1:/Table/72/{1/6-2}] removing replica r122/1 I190903 18:32:13.901555 77838 storage/replica_destroy.go:152 [n1,replicaGC,s1,r122/1:/Table/72/{1/6-2}] removed 5 (0+5) keys in 2ms [clear=0ms commit=2ms] I190903 18:32:13.908094 276 storage/compactor/compactor.go:386 [n1,s1,compactor] processed compaction #1-39/39 (/System/tsd-/Max) for 93 KiB in 0.0s I190903 18:32:13.962176 78022 storage/replica_command.go:1430 [n2,replicate,s2,r117/3:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r117:/Table/72/1/{3/5-5} [(n1,s1):1, (n2,s2):3, next=4, gen=69] I190903 18:32:13.969135 78022 storage/replica_raft.go:291 [n2,s2,r117/3:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):3] next=4 I190903 18:32:13.977806 78054 storage/store.go:2593 [n1,replicaGC,s1,r117/1:/Table/72/1/{3/5-5}] removing replica r117/1 I190903 18:32:13.978141 78054 storage/replica_destroy.go:152 [n1,replicaGC,s1,r117/1:/Table/72/1/{3/5-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:14.093829 78420 storage/replica_command.go:1430 [n1,replicate,s1,r121/3:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r121:/Table/72/1/{5/9-6} [(n2,s2):2, (n1,s1):3, next=4, gen=73] I190903 18:32:14.117409 78420 storage/replica_raft.go:291 [n1,s1,r121/3:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:14.128630 78418 storage/store.go:2593 [n2,replicaGC,s2,r121/2:/Table/72/1/{5/9-6}] removing replica r121/2 I190903 18:32:14.129108 78418 storage/replica_destroy.go:152 [n2,replicaGC,s2,r121/2:/Table/72/1/{5/9-6}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:14.139123 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.159532 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:14.164357 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.168765 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:14.171634 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.175950 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:14.180823 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:14.188408 78671 storage/replica_command.go:1430 [n1,replicate,s1,r116/3:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r116:/Table/72/1/3/{4-5} [(n2,s2):2, (n1,s1):3, next=4, gen=68] I190903 18:32:14.188854 65871 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.191713 78671 storage/replica_raft.go:291 [n1,s1,r116/3:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 === RUN TestRepartitioning/multi_col_list_partitioning_-_DEFAULT_DEFAULT/multi_col_list_partitioning_-_DEFAULT I190903 18:32:14.202120 78756 storage/store.go:2593 [n2,replicaGC,s2,r116/2:/Table/72/1/3/{4-5}] removing replica r116/2 I190903 18:32:14.202579 78756 storage/replica_destroy.go:152 [n2,replicaGC,s2,r116/2:/Table/72/1/3/{4-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:14.215979 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=5 mtime=2019-09-03 18:32:14.201707646 +0000 UTC I190903 18:32:14.232854 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 71, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning - DEFAULT DEFAULT"]} I190903 18:32:14.241499 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 4}] I190903 18:32:14.301245 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=6 mtime=2019-09-03 18:32:14.301071189 +0000 UTC I190903 18:32:14.319730 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 73, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:14.334997 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 74, info: {TableName:data.public."multi col list partitioning - DEFAULT DEFAULT" Statement:CREATE TABLE "multi col list partitioning - DEFAULT DEFAULT" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT)), PARTITION pd VALUES IN ((DEFAULT, DEFAULT))) User:root} I190903 18:32:14.337027 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:14.340767 80062 storage/replica_command.go:284 [n1,split,s1,r123/1:/{Table/72/2-Max}] initiating a split of this range at key /Table/74 [r86] (zone config) I190903 18:32:14.356445 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:14.358446 80170 storage/replica_command.go:284 [n1,split,s1,r86/1:/{Table/74-Max}] initiating a split of this range at key /Table/74/1/3/4 [r87] (zone config) I190903 18:32:14.368648 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:14.371897 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:14.372443 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:14.378467 80207 storage/replica_command.go:284 [n1,split,s1,r87/1:/{Table/74/1/3…-Max}] initiating a split of this range at key /Table/74/1/3/5 [r88] (zone config) I190903 18:32:14.390700 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:14.391574 78462 storage/replica_command.go:284 [n1,split,s1,r88/1:/{Table/74/1/3…-Max}] initiating a split of this range at key /Table/74/1/5/7 [r89] (zone config) I190903 18:32:14.401498 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:14.403620 80814 storage/replica_command.go:284 [n1,split,s1,r88/1:/Table/74/1/{3/5-5/7}] initiating a split of this range at key /Table/74/1/5 [r90] (zone config) I190903 18:32:14.416175 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION pd OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:14.416358 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:14.417010 81305 storage/replica_command.go:284 [n1,split,s1,r86/1:/Table/74{-/1/3/4}] initiating a split of this range at key /Table/74/1 [r91] (zone config) I190903 18:32:14.417575 80435 storage/replica_command.go:1430 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r88:/Table/74/1/{3/5-5} [(n1,s1):1, next=2, gen=77] I190903 18:32:14.431192 81599 storage/replica_command.go:284 [n1,split,s1,r89/1:/{Table/74/1/5…-Max}] initiating a split of this range at key /Table/74/1/5/8 [r92] (zone config) I190903 18:32:14.453644 81803 storage/replica_command.go:284 [n1,split,s1,r92/1:/{Table/74/1/5…-Max}] initiating a split of this range at key /Table/74/1/5/9 [r93] (zone config) I190903 18:32:14.456304 80435 storage/replica_raft.go:291 [n1,s1,r88/1:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:14.464273 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.469418 80435 storage/store_snapshot.go:995 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] sending LEARNER snapshot a306de70 at applied index 20 I190903 18:32:14.471034 80435 storage/store_snapshot.go:1038 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:14.472243 81767 storage/replica_raftstorage.go:793 [n2,s2,r88/2:{-}] applying LEARNER snapshot [id=a306de70 index=20] I190903 18:32:14.473310 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.473577 81767 storage/replica_raftstorage.go:814 [n2,s2,r88/2:/Table/74/1/{3/5-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=a306de70 index=20] I190903 18:32:14.476871 80435 storage/replica_command.go:1430 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r88:/Table/74/1/{3/5-5} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=78] I190903 18:32:14.479306 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.481649 80435 storage/replica_raft.go:291 [n1,s1,r88/1:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:14.482170 80435 storage/replica_command.go:1430 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r88:/Table/74/1/{3/5-5} [(n1,s1):1, (n2,s2):2, next=3, gen=79] I190903 18:32:14.482970 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.485528 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:14.488881 1646 storage/compactor/compactor.go:370 [n2,s2,compactor] processing compaction #1/33 (/System/tsd-/System/"tse") for 470 KiB (reasons: size=false used=true avail=false) I190903 18:32:14.489004 1646 storage/compactor/compactor.go:386 [n2,s2,compactor] processed compaction #1/33 (/System/tsd-/System/"tse") for 470 KiB in 0.0s I190903 18:32:14.489845 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.490818 81783 storage/replica_command.go:284 [n1,split,s1,r93/1:/{Table/74/1/5…-Max}] initiating a split of this range at key /Table/74/1/6 [r94] (zone config) I190903 18:32:14.496063 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.496167 80435 storage/replica_raft.go:291 [n1,s1,r88/1:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:14.496215 80435 storage/replica_raft.go:304 [n1,s1,r88/1:/Table/74/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.497085 80435 storage/replica_command.go:979 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r88 failed: storage/replica_raft.go:305: [n1,s1,r88/1:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:14.497514 80435 storage/queue.go:1027 [n1,replicate,s1,r88/1:/Table/74/1/{3/5-5}] change replicas of r88 failed: storage/replica_raft.go:305: [n1,s1,r88/1:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.499597 81897 storage/replica_command.go:1430 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r91:/Table/74/1{-/3/4} [(n1,s1):1, next=2, gen=75] I190903 18:32:14.502850 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:14.510305 81990 storage/replica_command.go:284 [n1,split,s1,r94/1:/{Table/74/1/6-Max}] initiating a split of this range at key /Table/74/2 [r95] (zone config) I190903 18:32:14.516312 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.516591 81897 storage/replica_raft.go:291 [n1,s1,r91/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:14.520881 81897 storage/store_snapshot.go:995 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] sending LEARNER snapshot 84428e9d at applied index 14 I190903 18:32:14.521038 81897 storage/store_snapshot.go:1038 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.521119 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:14.523057 81891 storage/replica_raftstorage.go:793 [n2,s2,r91/2:{-}] applying LEARNER snapshot [id=84428e9d index=14] I190903 18:32:14.524648 81891 storage/replica_raftstorage.go:814 [n2,s2,r91/2:/Table/74/1{-/3/4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=84428e9d index=14] I190903 18:32:14.525699 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.536822 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:14.536828 81897 storage/replica_command.go:1430 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r91:/Table/74/1{-/3/4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=76] I190903 18:32:14.538977 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.540299 81897 storage/replica_raft.go:291 [n1,s1,r91/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:14.540837 81897 storage/replica_command.go:1430 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r91:/Table/74/1{-/3/4} [(n1,s1):1, (n2,s2):2, next=3, gen=77] I190903 18:32:14.541097 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.548121 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.554492 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:14.570621 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.578453 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.582292 81897 storage/replica_raft.go:291 [n1,s1,r91/1:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:14.582333 81897 storage/replica_raft.go:304 [n1,s1,r91/1:/Table/74/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.583492 81897 storage/replica_command.go:979 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r91 failed: storage/replica_raft.go:305: [n1,s1,r91/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:14.583523 81897 storage/queue.go:1027 [n1,replicate,s1,r91/1:/Table/74/1{-/3/4}] change replicas of r91 failed: storage/replica_raft.go:305: [n1,s1,r91/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.584786 82090 storage/replica_command.go:1430 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r89:/Table/74/1/5/{7-8} [(n1,s1):1, next=2, gen=77] I190903 18:32:14.586403 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.588620 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.593859 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.596103 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.598171 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:14.598888 82090 storage/replica_raft.go:291 [n1,s1,r89/1:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:14.600950 82090 storage/store_snapshot.go:995 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] sending LEARNER snapshot 392bb194 at applied index 18 I190903 18:32:14.601112 82090 storage/store_snapshot.go:1038 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.601187 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:14.602574 82279 storage/replica_raftstorage.go:793 [n2,s2,r89/2:{-}] applying LEARNER snapshot [id=392bb194 index=18] I190903 18:32:14.611494 82279 storage/replica_raftstorage.go:814 [n2,s2,r89/2:/Table/74/1/5/{7-8}] applied LEARNER snapshot [total=9ms ingestion=4@8ms id=392bb194 index=18] I190903 18:32:14.618557 82090 storage/replica_command.go:1430 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] change replicas (add [(n2,s2):2] remove []): existing descriptor r89:/Table/74/1/5/{7-8} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=78] I190903 18:32:14.622931 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.625907 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.628716 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:14.632458 82090 storage/replica_raft.go:291 [n1,s1,r89/1:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:14.632772 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.633768 82090 storage/replica_command.go:1430 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r89:/Table/74/1/5/{7-8} [(n1,s1):1, (n2,s2):2, next=3, gen=79] I190903 18:32:14.640233 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:14.642173 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:14.643681 82090 storage/replica_raft.go:291 [n1,s1,r89/1:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:14.643715 82090 storage/replica_raft.go:304 [n1,s1,r89/1:/Table/74/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.646982 82090 storage/replica_command.go:979 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r89 failed: storage/replica_raft.go:305: [n1,s1,r89/1:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:14.647752 82090 storage/queue.go:1027 [n1,replicate,s1,r89/1:/Table/74/1/5/{7-8}] change replicas of r89 failed: storage/replica_raft.go:305: [n1,s1,r89/1:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.648894 82257 storage/replica_command.go:1430 [n1,replicate,s1,r86/1:/Table/74{-/1}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r86:/Table/74{-/1} [(n1,s1):1, next=2, gen=75] I190903 18:32:14.650263 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:14.654400 82257 storage/replica_raft.go:291 [n1,s1,r86/1:/Table/74{-/1}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:32:14.658251 82257 storage/store_snapshot.go:995 [n1,replicate,s1,r86/1:/Table/74{-/1}] sending LEARNER snapshot 5ef27709 at applied index 22 I190903 18:32:14.658392 82257 storage/store_snapshot.go:1038 [n1,replicate,s1,r86/1:/Table/74{-/1}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.659693 82436 storage/replica_raftstorage.go:793 [n3,s3,r86/2:{-}] applying LEARNER snapshot [id=5ef27709 index=22] I190903 18:32:14.661071 82436 storage/replica_raftstorage.go:814 [n3,s3,r86/2:/Table/74{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5ef27709 index=22] I190903 18:32:14.661888 82257 storage/replica_command.go:1430 [n1,replicate,s1,r86/1:/Table/74{-/1}] change replicas (add [(n3,s3):2] remove []): existing descriptor r86:/Table/74{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=76] I190903 18:32:14.662382 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:14.664527 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.668049 82257 storage/replica_raft.go:291 [n1,s1,r86/1:/Table/74{-/1}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:32:14.668572 82257 storage/replica_command.go:1430 [n1,replicate,s1,r86/1:/Table/74{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r86:/Table/74{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=77] I190903 18:32:14.684016 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:14.684934 82257 storage/replica_raft.go:291 [n1,s1,r86/1:/Table/74{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:14.684968 82257 storage/replica_raft.go:304 [n1,s1,r86/1:/Table/74{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:14.686475 82257 storage/replica_command.go:979 [n1,replicate,s1,r86/1:/Table/74{-/1}] could not promote [] to voter, rolling back: change replicas of r86 failed: storage/replica_raft.go:305: [n1,s1,r86/1:/Table/74{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:14.686522 82257 storage/queue.go:1027 [n1,replicate,s1,r86/1:/Table/74{-/1}] change replicas of r86 failed: storage/replica_raft.go:305: [n1,s1,r86/1:/Table/74{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:14.686947 82040 storage/replica_command.go:1430 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r92:/Table/74/1/5/{8-9} [(n1,s1):1, next=2, gen=78] I190903 18:32:14.687026 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.690699 82040 storage/replica_raft.go:291 [n1,s1,r92/1:/Table/74/1/5/{8-9}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:32:14.692110 82040 storage/store_snapshot.go:995 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] sending LEARNER snapshot 81a02b32 at applied index 18 I190903 18:32:14.692258 82040 storage/store_snapshot.go:1038 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.695518 81986 storage/replica_raftstorage.go:793 [n3,s3,r92/2:{-}] applying LEARNER snapshot [id=81a02b32 index=18] I190903 18:32:14.702983 81986 storage/replica_raftstorage.go:814 [n3,s3,r92/2:/Table/74/1/5/{8-9}] applied LEARNER snapshot [total=7ms ingestion=4@7ms id=81a02b32 index=18] I190903 18:32:14.703635 82040 storage/replica_command.go:1430 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] change replicas (add [(n3,s3):2] remove []): existing descriptor r92:/Table/74/1/5/{8-9} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=79] I190903 18:32:14.713497 82040 storage/replica_raft.go:291 [n1,s1,r92/1:/Table/74/1/5/{8-9}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:32:14.714011 82040 storage/replica_command.go:1430 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r92:/Table/74/1/5/{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=80] I190903 18:32:14.725682 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.728043 82040 storage/replica_raft.go:291 [n1,s1,r92/1:/Table/74/1/5/{8-9}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:14.728091 82040 storage/replica_raft.go:304 [n1,s1,r92/1:/Table/74/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:14.729073 82040 storage/replica_command.go:979 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r92 failed: storage/replica_raft.go:305: [n1,s1,r92/1:/Table/74/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:14.729130 82040 storage/queue.go:1027 [n1,replicate,s1,r92/1:/Table/74/1/5/{8-9}] change replicas of r92 failed: storage/replica_raft.go:305: [n1,s1,r92/1:/Table/74/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:14.729747 82646 storage/replica_command.go:1430 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r95:/{Table/74/2-Max} [(n1,s1):1, next=2, gen=80] I190903 18:32:14.734037 82646 storage/replica_raft.go:291 [n1,s1,r95/1:/{Table/74/2-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:14.735176 82646 storage/store_snapshot.go:995 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] sending LEARNER snapshot 67e89102 at applied index 15 I190903 18:32:14.735329 82646 storage/store_snapshot.go:1038 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.739938 82677 storage/replica_raftstorage.go:793 [n2,s2,r95/2:{-}] applying LEARNER snapshot [id=67e89102 index=15] I190903 18:32:14.741216 82677 storage/replica_raftstorage.go:814 [n2,s2,r95/2:/{Table/74/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=67e89102 index=15] I190903 18:32:14.745533 82646 storage/replica_command.go:1430 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r95:/{Table/74/2-Max} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=81] I190903 18:32:14.750567 82646 storage/replica_raft.go:291 [n1,s1,r95/1:/{Table/74/2-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:14.751114 82646 storage/replica_command.go:1430 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r95:/{Table/74/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=82] I190903 18:32:14.765179 82646 storage/replica_raft.go:291 [n1,s1,r95/1:/{Table/74/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:14.765233 82646 storage/replica_raft.go:304 [n1,s1,r95/1:/{Table/74/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.766082 82646 storage/replica_command.go:979 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] could not promote [] to voter, rolling back: change replicas of r95 failed: storage/replica_raft.go:305: [n1,s1,r95/1:/{Table/74/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:14.766127 82646 storage/queue.go:1027 [n1,replicate,s1,r95/1:/{Table/74/2-Max}] change replicas of r95 failed: storage/replica_raft.go:305: [n1,s1,r95/1:/{Table/74/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.782348 82562 storage/replica_command.go:1430 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r94:/Table/74/{1/6-2} [(n1,s1):1, next=2, gen=80] E190903 18:32:14.783070 82013 storage/queue.go:1027 [n2,replicate,s2,r91/2:/Table/74/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:21] E190903 18:32:14.784635 82015 storage/queue.go:1027 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:14.789840 82562 storage/replica_raft.go:291 [n1,s1,r94/1:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:14.790922 82562 storage/store_snapshot.go:995 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] sending LEARNER snapshot 69f74815 at applied index 18 I190903 18:32:14.791071 82562 storage/store_snapshot.go:1038 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.792286 82643 storage/replica_raftstorage.go:793 [n2,s2,r94/2:{-}] applying LEARNER snapshot [id=69f74815 index=18] I190903 18:32:14.793633 82643 storage/replica_raftstorage.go:814 [n2,s2,r94/2:/Table/74/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=69f74815 index=18] I190903 18:32:14.794336 82562 storage/replica_command.go:1430 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r94:/Table/74/{1/6-2} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=81] I190903 18:32:14.796742 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:14.799156 82562 storage/replica_raft.go:291 [n1,s1,r94/1:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:14.800681 82562 storage/replica_command.go:1430 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r94:/Table/74/{1/6-2} [(n1,s1):1, (n2,s2):2, next=3, gen=82] I190903 18:32:14.806260 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:14.813040 82562 storage/replica_raft.go:291 [n1,s1,r94/1:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:14.813086 82562 storage/replica_raft.go:304 [n1,s1,r94/1:/Table/74/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.813802 82562 storage/replica_command.go:979 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r94 failed: storage/replica_raft.go:305: [n1,s1,r94/1:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:14.817873 82562 storage/queue.go:1027 [n1,replicate,s1,r94/1:/Table/74/{1/6-2}] change replicas of r94 failed: storage/replica_raft.go:305: [n1,s1,r94/1:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:14.825272 82986 storage/replica_command.go:1430 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r123:/Table/7{2/2-4} [(n1,s1):1, next=3, gen=73] E190903 18:32:14.827387 82945 storage/queue.go:1027 [n3,replicate,s3,r86/2:/Table/74{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 18:32:14.830702 82986 storage/replica_raft.go:291 [n1,s1,r123/1:/Table/7{2/2-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n3,s3):3LEARNER] next=4 I190903 18:32:14.832610 82986 storage/store_snapshot.go:995 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] sending LEARNER snapshot 19062c1c at applied index 29 I190903 18:32:14.835494 82986 storage/store_snapshot.go:1038 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:14.840265 82990 storage/replica_raftstorage.go:793 [n3,s3,r123/3:{-}] applying LEARNER snapshot [id=19062c1c index=29] I190903 18:32:14.850274 82990 storage/replica_raftstorage.go:814 [n3,s3,r123/3:/Table/7{2/2-4}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=19062c1c index=29] I190903 18:32:14.850981 82986 storage/replica_command.go:1430 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r123:/Table/7{2/2-4} [(n1,s1):1, (n3,s3):3LEARNER, next=4, gen=74] I190903 18:32:14.858293 82986 storage/replica_raft.go:291 [n1,s1,r123/1:/Table/7{2/2-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 18:32:14.858964 82986 storage/replica_command.go:1430 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r123:/Table/7{2/2-4} [(n1,s1):1, (n3,s3):3, next=4, gen=75] I190903 18:32:14.876602 82986 storage/replica_raft.go:291 [n1,s1,r123/1:/Table/7{2/2-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 E190903 18:32:14.877519 82986 storage/replica_raft.go:304 [n1,s1,r123/1:/Table/7{2/2-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:14.878875 82986 storage/replica_command.go:979 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] could not promote [] to voter, rolling back: change replicas of r123 failed: storage/replica_raft.go:305: [n1,s1,r123/1:/Table/7{2/2-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:14.878948 82986 storage/queue.go:1027 [n1,replicate,s1,r123/1:/Table/7{2/2-4}] change replicas of r123 failed: storage/replica_raft.go:305: [n1,s1,r123/1:/Table/7{2/2-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:14.884747 82975 storage/replica_command.go:1430 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r92:/Table/74/1/5/{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=80] I190903 18:32:14.896547 82975 storage/replica_raft.go:291 [n3,s3,r92/2:/Table/74/1/5/{8-9}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:14.902017 83224 storage/queue.go:1027 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:14.930703 83322 storage/replica_command.go:1430 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r88:/Table/74/1/{3/5-5} [(n1,s1):1, (n2,s2):2, next=3, gen=79] I190903 18:32:14.942930 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:14.962188 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:14.964443 83214 storage/store.go:2593 [n1,replicaGC,s1,r92/1:/Table/74/1/5/{8-9}] removing replica r92/1 I190903 18:32:14.967092 83214 storage/replica_destroy.go:152 [n1,replicaGC,s1,r92/1:/Table/74/1/5/{8-9}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:14.968791 83322 storage/replica_raft.go:291 [n2,s2,r88/2:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:14.972907 83440 storage/replica_command.go:1430 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r89:/Table/74/1/5/{7-8} [(n1,s1):1, (n2,s2):2, next=3, gen=79] I190903 18:32:14.997813 83440 storage/replica_raft.go:291 [n2,s2,r89/2:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:14.999426 83427 storage/replica_command.go:1430 [n2,replicate,s2,r95/2:/{Table/74/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r95:/{Table/74/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=82] I190903 18:32:15.011997 83427 storage/replica_raft.go:291 [n2,s2,r95/2:/{Table/74/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:15.029903 83406 storage/store.go:2593 [n1,replicaGC,s1,r88/1:/Table/74/1/{3/5-5}] removing replica r88/1 I190903 18:32:15.030306 83406 storage/replica_destroy.go:152 [n1,replicaGC,s1,r88/1:/Table/74/1/{3/5-5}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:15.030887 83662 storage/store.go:2593 [n1,replicaGC,s1,r89/1:/Table/74/1/5/{7-8}] removing replica r89/1 I190903 18:32:15.031269 83662 storage/replica_destroy.go:152 [n1,replicaGC,s1,r89/1:/Table/74/1/5/{7-8}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:15.044118 83573 storage/replica_command.go:1430 [n3,replicate,s3,r86/2:/Table/74{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r86:/Table/74{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=77] I190903 18:32:15.072878 83573 storage/replica_raft.go:291 [n3,s3,r86/2:/Table/74{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 18:32:15.080994 83644 storage/store.go:2593 [n1,replicaGC,s1,r95/1:/{Table/74/2-Max}] removing replica r95/1 I190903 18:32:15.081224 83644 storage/replica_destroy.go:152 [n1,replicaGC,s1,r95/1:/{Table/74/2-Max}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:15.138878 83963 storage/store.go:2593 [n1,replicaGC,s1,r86/1:/Table/74{-/1}] removing replica r86/1 I190903 18:32:15.139197 83963 storage/replica_destroy.go:152 [n1,replicaGC,s1,r86/1:/Table/74{-/1}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:15.143538 83984 storage/replica_command.go:1430 [n2,replicate,s2,r91/2:/Table/74/1{-/3/4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r91:/Table/74/1{-/3/4} [(n1,s1):1, (n2,s2):2, next=3, gen=77] I190903 18:32:15.158038 83984 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:15.216589 84014 storage/store.go:2593 [n1,replicaGC,s1,r91/1:/Table/74/1{-/3/4}] removing replica r91/1 I190903 18:32:15.216889 84014 storage/replica_destroy.go:152 [n1,replicaGC,s1,r91/1:/Table/74/1{-/3/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:15.279893 84395 storage/queue.go:1027 [n3,replicate,s3,r123/3:/Table/7{2/2-4}] no removable replicas from range that needs a removal: [1:0, 3*:36] I190903 18:32:15.316650 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:15.320692 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:15.324175 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:15.326044 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:15.492959 84977 storage/replica_command.go:1430 [n3,replicate,s3,r123/3:/Table/7{2/2-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r123:/Table/7{2/2-4} [(n1,s1):1, (n3,s3):3, next=4, gen=75] I190903 18:32:15.503047 84977 storage/replica_raft.go:291 [n3,s3,r123/3:/Table/7{2/2-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 I190903 18:32:15.566244 84998 storage/store.go:2593 [n1,replicaGC,s1,r123/1:/Table/7{2/2-4}] removing replica r123/1 I190903 18:32:15.566634 84998 storage/replica_destroy.go:152 [n1,replicaGC,s1,r123/1:/Table/7{2/2-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:15.590708 85248 storage/queue.go:1027 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:15.816922 2356 storage/compactor/compactor.go:325 [n3,s3,compactor] purging suggested compaction for range /System/tsd - /System/"tse" that contains live data I190903 18:32:15.865812 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:15.869387 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:15.874662 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:16.117865 86701 storage/replica_command.go:1430 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r94:/Table/74/{1/6-2} [(n1,s1):1, (n2,s2):2, next=3, gen=82] I190903 18:32:16.132163 86701 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:16.140837 86416 storage/store.go:2593 [n1,replicaGC,s1,r94/1:/Table/74/{1/6-2}] removing replica r94/1 I190903 18:32:16.141226 86416 storage/replica_destroy.go:152 [n1,replicaGC,s1,r94/1:/Table/74/{1/6-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:16.796043 281 server/status/runtime.go:498 [n1] runtime stats: 238 MiB RSS, 946 goroutines, 65 MiB/34 MiB/138 MiB GO alloc/idle/total, 50 MiB/100 MiB CGO alloc/total, 11609.8 CGO/sec, 89.9/10.3 %(u/s)time, 1.0 %gc (15x), 104 MiB/104 MiB (r/w)net I190903 18:32:16.924926 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:16.929908 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:16.931766 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:16.935894 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:16.937556 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:16.945650 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:16.950682 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:16.953913 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 18:32:16.966132 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=74 (multi col list partitioning - DEFAULT) version=2 mtime=2019-09-03 18:32:16.957129364 +0000 UTC I190903 18:32:17.034324 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 1}] I190903 18:32:17.071575 89044 sql/lease.go:923 new lease: 74("multi col list partitioning - DEFAULT") ver=2:1567535823.758866016,0, refcount=0 I190903 18:32:17.092868 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=74 (multi col list partitioning - DEFAULT) version=3 mtime=2019-09-03 18:32:17.092685258 +0000 UTC I190903 18:32:17.105890 88897 sql/lease.go:923 new lease: 74("multi col list partitioning - DEFAULT") ver=3:1567535848.567197620,0, refcount=0 I190903 18:32:17.109745 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 2}] I190903 18:32:17.150578 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=74 (multi col list partitioning - DEFAULT) version=4 mtime=2019-09-03 18:32:17.14841373 +0000 UTC I190903 18:32:17.160579 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 74, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:ALTER TABLE "multi col list partitioning - DEFAULT" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:17.163671 89836 sql/lease.go:923 new lease: 74("multi col list partitioning - DEFAULT") ver=4:1567535848.568197620,0, refcount=0 I190903 18:32:17.164472 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 3}] I190903 18:32:17.166686 90019 storage/replica_command.go:1430 [n2,s2,r91/2:/Table/74/1{-/3/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r91:/Table/74/1{-/3/4} [(n2,s2):2, next=3, gen=78] I190903 18:32:17.167940 90070 storage/replica_command.go:609 [n2,merge,s2,r94/2:/Table/74/{1/6-2}] initiating a merge of r95:/{Table/74/2-Max} [(n2,s2):2, next=3, gen=83] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:17.192004 90019 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:17.195622 90019 storage/store_snapshot.go:995 [n2,s2,r91/2:/Table/74/1{-/3/4}] sending LEARNER snapshot 74ce1eec at applied index 33 I190903 18:32:17.195838 90019 storage/store_snapshot.go:1038 [n2,s2,r91/2:/Table/74/1{-/3/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:17.196906 90245 storage/replica_raftstorage.go:793 [n3,s3,r91/3:{-}] applying LEARNER snapshot [id=74ce1eec index=33] I190903 18:32:17.196917 1553 storage/store.go:2593 [n2,s2,r94/2:/Table/74/{1/6-2}] removing replica r95/2 I190903 18:32:17.208104 90245 storage/replica_raftstorage.go:814 [n3,s3,r91/3:/Table/74/1{-/3/4}] applied LEARNER snapshot [total=11ms ingestion=4@10ms id=74ce1eec index=33] I190903 18:32:17.212406 90019 storage/replica_command.go:1430 [n2,s2,r91/2:/Table/74/1{-/3/4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r91:/Table/74/1{-/3/4} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=79] I190903 18:32:17.223283 1651 server/status/runtime.go:498 [n2] runtime stats: 242 MiB RSS, 954 goroutines, 54 MiB/42 MiB/138 MiB GO alloc/idle/total, 50 MiB/102 MiB CGO alloc/total, 11244.8 CGO/sec, 88.3/10.1 %(u/s)time, 1.2 %gc (15x), 105 MiB/105 MiB (r/w)net I190903 18:32:17.236779 90019 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:17.238430 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:17.241374 90461 storage/replica_command.go:284 [n2,split,s2,r94/2:/{Table/74/1/6-Max}] initiating a split of this range at key /Table/74/2 [r124] (zone config) I190903 18:32:17.272428 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:17.286524 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} E190903 18:32:17.288254 90976 storage/queue.go:1027 [n3,replicate,s3,r91/3:/Table/74/1{-/3/4}] no removable replicas from range that needs a removal: [2:0, 3*:38] I190903 18:32:17.303139 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:17.330601 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:17.332083 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:17.349032 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:17.388414 92073 storage/replica_command.go:1430 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r94:/Table/74/{1/6-2} [(n2,s2):2, next=3, gen=85] I190903 18:32:17.394704 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:17.411812 91847 storage/replica_command.go:1430 [n3,s3,r91/3:/Table/74/1{-/3/4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r91:/Table/74/1{-/3/4} [(n2,s2):2, (n3,s3):3, next=4, gen=80] I190903 18:32:17.414352 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:17.419581 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:17.422116 91847 storage/replica_raft.go:291 [n3,s3,r91/3:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:17.423806 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:17.425696 92073 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:17.426212 90035 storage/replica_command.go:609 [n3,merge,s3,r86/2:/Table/74{-/1}] initiating a merge of r91:/Table/74/1{-/3/4} [(n3,s3):3, next=4, gen=81] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:17.428374 92073 storage/store_snapshot.go:995 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] sending LEARNER snapshot 46f446c5 at applied index 43 I190903 18:32:17.428825 92073 storage/store_snapshot.go:1038 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:17.430808 91213 storage/replica_raftstorage.go:793 [n1,s1,r94/3:{-}] applying LEARNER snapshot [id=46f446c5 index=43] I190903 18:32:17.433585 92033 storage/replica_command.go:1430 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r93:/Table/74/1/{5/9-6} [(n1,s1):1, next=2, gen=79] I190903 18:32:17.438425 92270 storage/store.go:2593 [n2,replicaGC,s2,r91/2:/Table/74/1{-/3/4}] removing replica r91/2 I190903 18:32:17.438961 92270 storage/replica_destroy.go:152 [n2,replicaGC,s2,r91/2:/Table/74/1{-/3/4}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:17.446614 91213 storage/replica_raftstorage.go:814 [n1,s1,r94/3:/Table/74/{1/6-2}] applied LEARNER snapshot [total=16ms ingestion=4@15ms id=46f446c5 index=43] I190903 18:32:17.447469 92073 storage/replica_command.go:1430 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] change replicas (add [(n1,s1):3] remove []): existing descriptor r94:/Table/74/{1/6-2} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=86] I190903 18:32:17.449139 92033 storage/replica_raft.go:291 [n1,s1,r93/1:/Table/74/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:17.450432 92033 storage/store_snapshot.go:995 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] sending LEARNER snapshot 29682e32 at applied index 20 I190903 18:32:17.450731 92033 storage/store_snapshot.go:1038 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:17.452021 92321 storage/replica_raftstorage.go:793 [n2,s2,r93/2:{-}] applying LEARNER snapshot [id=29682e32 index=20] I190903 18:32:17.454525 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:17.458735 92321 storage/replica_raftstorage.go:814 [n2,s2,r93/2:/Table/74/1/{5/9-6}] applied LEARNER snapshot [total=7ms ingestion=4@1ms id=29682e32 index=20] I190903 18:32:17.459269 92033 storage/replica_command.go:1430 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r93:/Table/74/1/{5/9-6} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=80] I190903 18:32:17.459835 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:17.463291 92033 storage/replica_raft.go:291 [n1,s1,r93/1:/Table/74/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:17.463924 92033 storage/replica_command.go:1430 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r93:/Table/74/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=81] I190903 18:32:17.464539 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:17.467896 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:17.469803 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:17.473835 92033 storage/replica_raft.go:291 [n1,s1,r93/1:/Table/74/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:17.473945 92033 storage/replica_raft.go:304 [n1,s1,r93/1:/Table/74/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:17.474344 90035 storage/replica_command.go:609 [n3,merge,s3,r86/2:/Table/74{-/1}] initiating a merge of r91:/Table/74/1{-/3/4} [(n3,s3):3, next=4, gen=81] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:17.474868 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:17.475061 92033 storage/replica_command.go:979 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r93 failed: storage/replica_raft.go:305: [n1,s1,r93/1:/Table/74/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:17.475089 92033 storage/queue.go:1027 [n1,replicate,s1,r93/1:/Table/74/1/{5/9-6}] change replicas of r93 failed: storage/replica_raft.go:305: [n1,s1,r93/1:/Table/74/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:17.475601 92398 storage/replica_command.go:1430 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r90:/Table/74/1/5{-/7} [(n1,s1):1, next=2, gen=77] I190903 18:32:17.475894 92073 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:17.480091 92398 storage/replica_raft.go:291 [n1,s1,r90/1:/Table/74/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:17.482076 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:17.482643 92398 storage/store_snapshot.go:995 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] sending LEARNER snapshot d77557c8 at applied index 15 I190903 18:32:17.486115 92398 storage/store_snapshot.go:1038 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:17.487143 92073 storage/replica_command.go:1430 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r94:/Table/74/{1/6-2} [(n2,s2):2, (n1,s1):3, next=4, gen=87] I190903 18:32:17.488245 92533 storage/replica_raftstorage.go:793 [n2,s2,r90/2:{-}] applying LEARNER snapshot [id=d77557c8 index=15] I190903 18:32:17.490134 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:17.490871 92533 storage/replica_raftstorage.go:814 [n2,s2,r90/2:/Table/74/1/5{-/7}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=d77557c8 index=15] I190903 18:32:17.491460 92398 storage/replica_command.go:1430 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r90:/Table/74/1/5{-/7} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=78] I190903 18:32:17.492320 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:17.497458 92398 storage/replica_raft.go:291 [n1,s1,r90/1:/Table/74/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:17.498290 2361 server/status/runtime.go:498 [n3] runtime stats: 244 MiB RSS, 973 goroutines, 59 MiB/37 MiB/138 MiB GO alloc/idle/total, 52 MiB/105 MiB CGO alloc/total, 11210.7 CGO/sec, 88.8/10.2 %(u/s)time, 1.2 %gc (15x), 105 MiB/105 MiB (r/w)net I190903 18:32:17.498952 92398 storage/replica_command.go:1430 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r90:/Table/74/1/5{-/7} [(n1,s1):1, (n2,s2):2, next=3, gen=79] I190903 18:32:17.499423 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:17.502176 2282 storage/store.go:2593 [n3,s3,r86/2:/Table/74{-/1}] removing replica r91/3 I190903 18:32:17.502653 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:17.505660 92073 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:17.505701 92073 storage/replica_raft.go:304 [n2,s2,r94/2:/Table/74/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:17.506365 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:17.508833 92073 storage/replica_command.go:979 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r94 failed: storage/replica_raft.go:305: [n2,s2,r94/2:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:17.508895 92073 storage/queue.go:1027 [n2,replicate,s2,r94/2:/Table/74/{1/6-2}] change replicas of r94 failed: storage/replica_raft.go:305: [n2,s2,r94/2:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:17.511511 92398 storage/replica_raft.go:291 [n1,s1,r90/1:/Table/74/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:17.511765 92398 storage/replica_raft.go:304 [n1,s1,r90/1:/Table/74/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:17.512732 92398 storage/replica_command.go:979 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r90 failed: storage/replica_raft.go:305: [n1,s1,r90/1:/Table/74/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:17.512789 92398 storage/queue.go:1027 [n1,replicate,s1,r90/1:/Table/74/1/5{-/7}] change replicas of r90 failed: storage/replica_raft.go:305: [n1,s1,r90/1:/Table/74/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:17.525566 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:17.532518 92711 storage/replica_command.go:1430 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r87:/Table/74/1/3/{4-5} [(n1,s1):1, next=2, gen=75] I190903 18:32:17.534345 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:17.537689 92711 storage/replica_raft.go:291 [n1,s1,r87/1:/Table/74/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 18:32:17.542475 92711 storage/store_snapshot.go:995 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] sending LEARNER snapshot cffab363 at applied index 20 I190903 18:32:17.542783 92711 storage/store_snapshot.go:1038 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:17.545902 92691 storage/replica_command.go:1430 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r92:/Table/74/1/5/{8-9} [(n3,s3):2, next=3, gen=81] I190903 18:32:17.548670 92742 storage/replica_raftstorage.go:793 [n2,s2,r87/2:{-}] applying LEARNER snapshot [id=cffab363 index=20] I190903 18:32:17.557955 92742 storage/replica_raftstorage.go:814 [n2,s2,r87/2:/Table/74/1/3/{4-5}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=cffab363 index=20] I190903 18:32:17.561761 92711 storage/replica_command.go:1430 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r87:/Table/74/1/3/{4-5} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=76] I190903 18:32:17.566099 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:17.566533 92711 storage/replica_raft.go:291 [n1,s1,r87/1:/Table/74/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 18:32:17.567244 92711 storage/replica_command.go:1430 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r87:/Table/74/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=77] I190903 18:32:17.581595 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:17.582034 92711 storage/replica_raft.go:291 [n1,s1,r87/1:/Table/74/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 18:32:17.582070 92711 storage/replica_raft.go:304 [n1,s1,r87/1:/Table/74/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:17.582226 92691 storage/replica_raft.go:291 [n3,s3,r92/2:/Table/74/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 18:32:17.582883 92711 storage/replica_command.go:979 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r87 failed: storage/replica_raft.go:305: [n1,s1,r87/1:/Table/74/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:17.585201 92711 storage/queue.go:1027 [n1,replicate,s1,r87/1:/Table/74/1/3/{4-5}] change replicas of r87 failed: storage/replica_raft.go:305: [n1,s1,r87/1:/Table/74/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:17.583445 92691 storage/store_snapshot.go:995 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] sending LEARNER snapshot 8b5d0d42 at applied index 36 I190903 18:32:17.586478 92691 storage/store_snapshot.go:1038 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:17.587826 92895 storage/replica_raftstorage.go:793 [n1,s1,r92/3:{-}] applying LEARNER snapshot [id=8b5d0d42 index=36] I190903 18:32:17.589121 92895 storage/replica_raftstorage.go:814 [n1,s1,r92/3:/Table/74/1/5/{8-9}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=8b5d0d42 index=36] I190903 18:32:17.589699 92691 storage/replica_command.go:1430 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] change replicas (add [(n1,s1):3] remove []): existing descriptor r92:/Table/74/1/5/{8-9} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=82] I190903 18:32:17.603901 92691 storage/replica_raft.go:291 [n3,s3,r92/2:/Table/74/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 18:32:17.604543 92691 storage/replica_command.go:1430 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r92:/Table/74/1/5/{8-9} [(n3,s3):2, (n1,s1):3, next=4, gen=83] I190903 18:32:17.619740 92691 storage/replica_raft.go:291 [n3,s3,r92/2:/Table/74/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 18:32:17.619780 92691 storage/replica_raft.go:304 [n3,s3,r92/2:/Table/74/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:17.620619 92691 storage/replica_command.go:979 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r92 failed: storage/replica_raft.go:305: [n3,s3,r92/2:/Table/74/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:17.620689 92691 storage/queue.go:1027 [n3,replicate,s3,r92/2:/Table/74/1/5/{8-9}] change replicas of r92 failed: storage/replica_raft.go:305: [n3,s3,r92/2:/Table/74/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:17.624277 92963 storage/replica_command.go:284 [n3,split,s3,r86/2:/Table/74{-/1/3/4}] initiating a split of this range at key /Table/74/1 [r66] (zone config) I190903 18:32:17.653252 93086 storage/replica_command.go:1430 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r66:/Table/74/1{-/3/4} [(n3,s3):1, next=2, gen=83] I190903 18:32:17.653728 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:17.664236 93086 storage/replica_raft.go:291 [n3,s3,r66/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:17.666065 93086 storage/store_snapshot.go:995 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] sending LEARNER snapshot 0e0474c3 at applied index 14 I190903 18:32:17.666366 93086 storage/store_snapshot.go:1038 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 19, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:17.669806 92862 storage/replica_raftstorage.go:793 [n1,s1,r66/2:{-}] applying LEARNER snapshot [id=0e0474c3 index=14] I190903 18:32:17.670603 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:17.677771 92862 storage/replica_raftstorage.go:814 [n1,s1,r66/2:/Table/74/1{-/3/4}] applied LEARNER snapshot [total=8ms ingestion=4@8ms id=0e0474c3 index=14] I190903 18:32:17.679013 93086 storage/replica_command.go:1430 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r66:/Table/74/1{-/3/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=84] I190903 18:32:17.684794 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 18:32:17.688242 93086 storage/replica_raft.go:291 [n3,s3,r66/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:17.689047 93086 storage/replica_command.go:1430 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r66:/Table/74/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=85] I190903 18:32:17.720358 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:17.730950 93086 storage/replica_raft.go:291 [n3,s3,r66/1:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:17.731016 93086 storage/replica_raft.go:304 [n3,s3,r66/1:/Table/74/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:17.738718 93086 storage/replica_command.go:979 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r66 failed: storage/replica_raft.go:305: [n3,s3,r66/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:17.738891 93086 storage/queue.go:1027 [n3,replicate,s3,r66/1:/Table/74/1{-/3/4}] change replicas of r66 failed: storage/replica_raft.go:305: [n3,s3,r66/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:17.751082 93498 storage/queue.go:1027 [n1,replicate,s1,r66/2:/Table/74/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:21] I190903 18:32:17.772127 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:17.841925 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 E190903 18:32:17.874380 93747 storage/queue.go:1027 [n2,replicate,s2,r87/2:/Table/74/1/3/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:17.979780 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:18.005163 94005 storage/replica_command.go:1430 [n2,replicate,s2,r87/2:/Table/74/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r87:/Table/74/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=77] I190903 18:32:18.014797 94005 storage/replica_raft.go:291 [n2,s2,r87/2:/Table/74/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:18.019249 94177 storage/replica_command.go:1430 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r88:/Table/74/1/{3/5-5} [(n2,s2):2, next=3, gen=80] E190903 18:32:18.019934 94230 storage/queue.go:1027 [n1,replicate,s1,r94/3:/Table/74/{1/6-2}] no removable replicas from range that needs a removal: [2:0, 3*:50] I190903 18:32:18.030324 94177 storage/replica_raft.go:291 [n2,s2,r88/2:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:18.031665 94177 storage/store_snapshot.go:995 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] sending LEARNER snapshot d4221f3e at applied index 38 I190903 18:32:18.031832 94177 storage/store_snapshot.go:1038 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:18.033087 93325 storage/replica_raftstorage.go:793 [n1,s1,r88/3:{-}] applying LEARNER snapshot [id=d4221f3e index=38] I190903 18:32:18.043597 93325 storage/replica_raftstorage.go:814 [n1,s1,r88/3:/Table/74/1/{3/5-5}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=d4221f3e index=38] I190903 18:32:18.044258 94177 storage/replica_command.go:1430 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r88:/Table/74/1/{3/5-5} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=81] I190903 18:32:18.060162 94177 storage/replica_raft.go:291 [n2,s2,r88/2:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:18.060707 94177 storage/replica_command.go:1430 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r88:/Table/74/1/{3/5-5} [(n2,s2):2, (n1,s1):3, next=4, gen=82] I190903 18:32:18.071535 94177 storage/replica_raft.go:291 [n2,s2,r88/2:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:18.071580 94177 storage/replica_raft.go:304 [n2,s2,r88/2:/Table/74/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:18.073396 94177 storage/replica_command.go:979 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r88 failed: storage/replica_raft.go:305: [n2,s2,r88/2:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:18.073488 94177 storage/queue.go:1027 [n2,replicate,s2,r88/2:/Table/74/1/{3/5-5}] change replicas of r88 failed: storage/replica_raft.go:305: [n2,s2,r88/2:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:18.086930 94232 storage/queue.go:1027 [n2,replicate,s2,r93/2:/Table/74/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:18.096729 93904 storage/store.go:2593 [n1,replicaGC,s1,r87/1:/Table/74/1/3/{4-5}] removing replica r87/1 I190903 18:32:18.098398 93904 storage/replica_destroy.go:152 [n1,replicaGC,s1,r87/1:/Table/74/1/3/{4-5}] removed 8 (0+8) keys in 2ms [clear=0ms commit=1ms] E190903 18:32:18.103880 94533 storage/queue.go:1027 [n2,replicate,s2,r90/2:/Table/74/1/5{-/7}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:18.122705 94555 storage/replica_command.go:1430 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r89:/Table/74/1/5/{7-8} [(n2,s2):2, next=3, gen=80] I190903 18:32:18.139995 94555 storage/replica_raft.go:291 [n2,s2,r89/2:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:18.142323 94555 storage/store_snapshot.go:995 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] sending LEARNER snapshot 13fe89b8 at applied index 36 I190903 18:32:18.143028 94555 storage/store_snapshot.go:1038 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:18.145055 94370 storage/replica_raftstorage.go:793 [n3,s3,r89/3:{-}] applying LEARNER snapshot [id=13fe89b8 index=36] I190903 18:32:18.158592 94370 storage/replica_raftstorage.go:814 [n3,s3,r89/3:/Table/74/1/5/{7-8}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=13fe89b8 index=36] I190903 18:32:18.159460 94555 storage/replica_command.go:1430 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] change replicas (add [(n3,s3):3] remove []): existing descriptor r89:/Table/74/1/5/{7-8} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=81] I190903 18:32:18.167306 94555 storage/replica_raft.go:291 [n2,s2,r89/2:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 18:32:18.167891 94555 storage/replica_command.go:1430 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r89:/Table/74/1/5/{7-8} [(n2,s2):2, (n3,s3):3, next=4, gen=82] I190903 18:32:18.183653 94555 storage/replica_raft.go:291 [n2,s2,r89/2:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 18:32:18.183690 94555 storage/replica_raft.go:304 [n2,s2,r89/2:/Table/74/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:18.185126 94555 storage/replica_command.go:979 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r89 failed: storage/replica_raft.go:305: [n2,s2,r89/2:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:18.185182 94555 storage/queue.go:1027 [n2,replicate,s2,r89/2:/Table/74/1/5/{7-8}] change replicas of r89 failed: storage/replica_raft.go:305: [n2,s2,r89/2:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:18.232738 94849 storage/queue.go:1027 [n1,replicate,s1,r92/3:/Table/74/1/5/{8-9}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 18:32:18.254326 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:18.261910 94993 storage/replica_command.go:1430 [n2,replicate,s2,r90/2:/Table/74/1/5{-/7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r90:/Table/74/1/5{-/7} [(n1,s1):1, (n2,s2):2, next=3, gen=79] I190903 18:32:18.272802 94993 storage/replica_raft.go:291 [n2,s2,r90/2:/Table/74/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:18.285678 95062 storage/store.go:2593 [n1,replicaGC,s1,r90/1:/Table/74/1/5{-/7}] removing replica r90/1 I190903 18:32:18.286005 95062 storage/replica_destroy.go:152 [n1,replicaGC,s1,r90/1:/Table/74/1/5{-/7}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.394233 95231 storage/replica_command.go:1430 [n1,replicate,s1,r66/2:/Table/74/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r66:/Table/74/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=85] I190903 18:32:18.400250 95231 storage/replica_raft.go:291 [n1,s1,r66/2:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:18.405043 95338 storage/replica_command.go:1430 [n1,replicate,s1,r88/3:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r88:/Table/74/1/{3/5-5} [(n2,s2):2, (n1,s1):3, next=4, gen=82] I190903 18:32:18.408694 95338 storage/replica_raft.go:291 [n1,s1,r88/3:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:18.409662 95428 storage/queue.go:1027 [n3,replicate,s3,r89/3:/Table/74/1/5/{7-8}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 18:32:18.416309 95053 storage/store.go:2593 [n3,replicaGC,s3,r66/1:/Table/74/1{-/3/4}] removing replica r66/1 I190903 18:32:18.416451 95277 storage/store.go:2593 [n2,replicaGC,s2,r88/2:/Table/74/1/{3/5-5}] removing replica r88/2 I190903 18:32:18.416730 95053 storage/replica_destroy.go:152 [n3,replicaGC,s3,r66/1:/Table/74/1{-/3/4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.416881 95277 storage/replica_destroy.go:152 [n2,replicaGC,s2,r88/2:/Table/74/1/{3/5-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.525242 95766 storage/replica_command.go:1430 [n1,replicate,s1,r94/3:/Table/74/{1/6-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r94:/Table/74/{1/6-2} [(n2,s2):2, (n1,s1):3, next=4, gen=87] I190903 18:32:18.528649 95766 storage/replica_raft.go:291 [n1,s1,r94/3:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:18.537385 95730 storage/store.go:2593 [n2,replicaGC,s2,r94/2:/Table/74/{1/6-2}] removing replica r94/2 I190903 18:32:18.537888 95730 storage/replica_destroy.go:152 [n2,replicaGC,s2,r94/2:/Table/74/{1/6-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.541178 95341 storage/replica_command.go:1430 [n2,replicate,s2,r93/2:/Table/74/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r93:/Table/74/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=81] I190903 18:32:18.549143 95341 storage/replica_raft.go:291 [n2,s2,r93/2:/Table/74/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 18:32:18.557407 95343 storage/store.go:2593 [n1,replicaGC,s1,r93/1:/Table/74/1/{5/9-6}] removing replica r93/1 I190903 18:32:18.557720 95343 storage/replica_destroy.go:152 [n1,replicaGC,s1,r93/1:/Table/74/1/{5/9-6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.758199 96067 storage/replica_command.go:1430 [n1,replicate,s1,r92/3:/Table/74/1/5/{8-9}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r92:/Table/74/1/5/{8-9} [(n3,s3):2, (n1,s1):3, next=4, gen=83] I190903 18:32:18.761609 96067 storage/replica_raft.go:291 [n1,s1,r92/3:/Table/74/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 18:32:18.770406 95695 storage/store.go:2593 [n3,replicaGC,s3,r92/2:/Table/74/1/5/{8-9}] removing replica r92/2 I190903 18:32:18.770892 95695 storage/replica_destroy.go:152 [n3,replicaGC,s3,r92/2:/Table/74/1/5/{8-9}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.855617 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 18:32:18.858800 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 18:32:18.860563 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 18:32:18.863537 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 18:32:18.869321 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 18:32:18.871260 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 18:32:18.875569 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:18.877293 78757 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) === RUN TestRepartitioning/single_col_range_partitioning/single_col_range_partitioning_-_MAXVALUE I190903 18:32:18.893263 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=74 (multi col list partitioning - DEFAULT) version=5 mtime=2019-09-03 18:32:18.880005857 +0000 UTC I190903 18:32:18.903585 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 73, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning - DEFAULT"]} I190903 18:32:18.906701 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 4}] I190903 18:32:18.909315 96843 storage/replica_command.go:1430 [n3,replicate,s3,r89/3:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r89:/Table/74/1/5/{7-8} [(n2,s2):2, (n3,s3):3, next=4, gen=82] I190903 18:32:18.915659 96843 storage/replica_raft.go:291 [n3,s3,r89/3:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:18.929524 97205 storage/store.go:2593 [n2,replicaGC,s2,r89/2:/Table/74/1/5/{7-8}] removing replica r89/2 I190903 18:32:18.929967 97205 storage/replica_destroy.go:152 [n2,replicaGC,s2,r89/2:/Table/74/1/5/{7-8}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:18.965984 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=74 (multi col list partitioning - DEFAULT) version=6 mtime=2019-09-03 18:32:18.965744154 +0000 UTC I190903 18:32:18.986109 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 75, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:18.986815 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:18.992303 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 76, info: {TableName:data.public."single col range partitioning" Statement:CREATE TABLE "single col range partitioning" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root} I190903 18:32:18.995345 98169 storage/replica_command.go:284 [n2,split,s2,r124/1:/{Table/74/2-Max}] initiating a split of this range at key /Table/76 [r125] (zone config) I190903 18:32:19.017024 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 76, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:19.025328 98616 storage/replica_command.go:284 [n2,split,s2,r125/1:/{Table/76-Max}] initiating a split of this range at key /Table/76/1 [r126] (zone config) I190903 18:32:19.026730 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:19.035904 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:19.036607 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.049104 98286 storage/replica_command.go:284 [n2,split,s2,r126/1:/{Table/76/1-Max}] initiating a split of this range at key /Table/76/1/3 [r127] (zone config) I190903 18:32:19.053402 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.063209 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.068280 99279 storage/replica_command.go:284 [n2,split,s2,r127/1:/{Table/76/1/3-Max}] initiating a split of this range at key /Table/76/1/4 [r128] (zone config) I190903 18:32:19.068639 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.096098 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.096846 99151 storage/replica_command.go:284 [n2,split,s2,r128/1:/{Table/76/1/4-Max}] initiating a split of this range at key /Table/76/2 [r129] (zone config) I190903 18:32:19.097421 99462 storage/replica_command.go:1430 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r127:/Table/76/1/{3-4} [(n2,s2):1, next=2, gen=89] I190903 18:32:19.102822 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.110754 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.113647 99462 storage/replica_raft.go:291 [n2,s2,r127/1:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:19.116274 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.118096 99462 storage/store_snapshot.go:995 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] sending LEARNER snapshot 04ac7f52 at applied index 19 I190903 18:32:19.118255 99462 storage/store_snapshot.go:1038 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:19.120161 99545 storage/replica_raftstorage.go:793 [n3,s3,r127/2:{-}] applying LEARNER snapshot [id=04ac7f52 index=19] I190903 18:32:19.121589 99545 storage/replica_raftstorage.go:814 [n3,s3,r127/2:/Table/76/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=04ac7f52 index=19] I190903 18:32:19.122125 99462 storage/replica_command.go:1430 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r127:/Table/76/1/{3-4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=90] I190903 18:32:19.128945 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.132569 99462 storage/replica_raft.go:291 [n2,s2,r127/1:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 18:32:19.133188 99462 storage/replica_command.go:1430 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r127:/Table/76/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=91] I190903 18:32:19.133708 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.138557 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.144548 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.150075 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.152430 99462 storage/replica_raft.go:291 [n2,s2,r127/1:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 18:32:19.152480 99462 storage/replica_raft.go:304 [n2,s2,r127/1:/Table/76/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:19.153390 99462 storage/replica_command.go:979 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r127 failed: storage/replica_raft.go:305: [n2,s2,r127/1:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:19.153464 99462 storage/queue.go:1027 [n2,replicate,s2,r127/1:/Table/76/1/{3-4}] change replicas of r127 failed: storage/replica_raft.go:305: [n2,s2,r127/1:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:19.155645 99591 storage/replica_command.go:1430 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r128:/Table/76/{1/4-2} [(n2,s2):1, next=2, gen=90] E190903 18:32:19.156835 99692 storage/queue.go:1027 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:19.172353 99591 storage/replica_raft.go:291 [n2,s2,r128/1:/Table/76/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:19.176297 99591 storage/store_snapshot.go:995 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] sending LEARNER snapshot eb11ebbf at applied index 17 I190903 18:32:19.176724 99591 storage/store_snapshot.go:1038 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:19.178218 99876 storage/replica_raftstorage.go:793 [n1,s1,r128/2:{-}] applying LEARNER snapshot [id=eb11ebbf index=17] I190903 18:32:19.192472 99876 storage/replica_raftstorage.go:814 [n1,s1,r128/2:/Table/76/{1/4-2}] applied LEARNER snapshot [total=14ms ingestion=4@14ms id=eb11ebbf index=17] I190903 18:32:19.193343 99591 storage/replica_command.go:1430 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r128:/Table/76/{1/4-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=91] I190903 18:32:19.214998 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.221182 99591 storage/replica_raft.go:291 [n2,s2,r128/1:/Table/76/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:19.224426 99591 storage/replica_command.go:1430 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r128:/Table/76/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=92] I190903 18:32:19.229152 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.234804 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.239684 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.259094 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.262399 99591 storage/replica_raft.go:291 [n2,s2,r128/1:/Table/76/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:19.262462 99591 storage/replica_raft.go:304 [n2,s2,r128/1:/Table/76/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:19.263549 99591 storage/replica_command.go:979 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r128 failed: storage/replica_raft.go:305: [n2,s2,r128/1:/Table/76/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:19.264081 99591 storage/queue.go:1027 [n2,replicate,s2,r128/1:/Table/76/{1/4-2}] change replicas of r128 failed: storage/replica_raft.go:305: [n2,s2,r128/1:/Table/76/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:19.272323 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.276649 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.284498 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.291688 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.305027 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.358407 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.362558 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.369431 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.403203 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.407455 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.411553 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.428905 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.432584 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.442577 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.446787 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.450594 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.461329 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.474561 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.480839 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.491034 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.494922 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.500569 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.504270 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.514652 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.528561 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.534298 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.541103 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.545241 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.548589 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.556279 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.560605 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.564182 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.576845 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.581211 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.586571 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.609388 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.613335 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.618104 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.656267 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.661132 101537 storage/replica_command.go:1430 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r127:/Table/76/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=91] I190903 18:32:19.662545 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.665546 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.669307 101537 storage/replica_raft.go:291 [n3,s3,r127/2:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 18:32:19.680203 101599 storage/store.go:2593 [n2,replicaGC,s2,r127/1:/Table/76/1/{3-4}] removing replica r127/1 I190903 18:32:19.680639 101599 storage/replica_destroy.go:152 [n2,replicaGC,s2,r127/1:/Table/76/1/{3-4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:19.738817 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.742047 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.753277 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 E190903 18:32:19.862578 102186 storage/queue.go:1027 [n1,replicate,s1,r128/2:/Table/76/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:19.896291 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:19.899838 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:19.902537 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:19.907455 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=76 (single col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 18:32:19.906052385 +0000 UTC I190903 18:32:19.918642 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 1}] I190903 18:32:19.919498 102175 storage/replica_command.go:1430 [n1,replicate,s1,r128/2:/Table/76/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r128:/Table/76/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=92] I190903 18:32:19.920813 102367 sql/lease.go:923 new lease: 76("single col range partitioning - MAXVALUE") ver=2:1567535850.623261132,0, refcount=0 I190903 18:32:19.923184 102175 storage/replica_raft.go:291 [n1,s1,r128/2:/Table/76/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:19.935000 102526 storage/store.go:2593 [n2,replicaGC,s2,r128/1:/Table/76/{1/4-2}] removing replica r128/1 I190903 18:32:19.935465 102526 storage/replica_destroy.go:152 [n2,replicaGC,s2,r128/1:/Table/76/{1/4-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:19.978107 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=76 (single col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 18:32:19.977934906 +0000 UTC I190903 18:32:19.982575 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:19.982829 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 2}] I190903 18:32:19.985824 102918 sql/lease.go:923 new lease: 76("single col range partitioning - MAXVALUE") ver=3:1567535852.952250112,0, refcount=0 I190903 18:32:20.010565 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=76 (single col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 18:32:20.008183569 +0000 UTC I190903 18:32:20.019574 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 76, info: {TableName:data.public."single col range partitioning - MAXVALUE" Statement:ALTER TABLE "single col range partitioning - MAXVALUE" PARTITION BY RANGE (a) (PARTITION p4 VALUES FROM (minvalue) TO (4), PARTITION p5 VALUES FROM (4) TO (5), PARTITION px VALUES FROM (5) TO (maxvalue)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:20.023072 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 3}] I190903 18:32:20.023196 103274 sql/lease.go:923 new lease: 76("single col range partitioning - MAXVALUE") ver=4:1567535852.953250112,0, refcount=0 I190903 18:32:20.060429 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} E190903 18:32:20.068346 103627 storage/queue.go:1027 [n2,merge,s2,r126/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 I190903 18:32:20.072798 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p5 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:20.075067 104099 storage/replica_command.go:284 [n1,split,s1,r128/2:/Table/76/{1/4-2}] initiating a split of this range at key /Table/76/1/5 [r96] (zone config) E190903 18:32:20.078588 104190 storage/queue.go:1027 [n2,merge,s2,r126/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 I190903 18:32:20.103148 104328 storage/replica_command.go:1430 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r128:/Table/76/1/{4-5} [(n1,s1):2, next=3, gen=94] I190903 18:32:20.109362 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION px OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:20.109461 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 18:32:20.113789 104589 storage/queue.go:1027 [n2,merge,s2,r126/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 I190903 18:32:20.116512 104328 storage/replica_raft.go:291 [n1,s1,r128/2:/Table/76/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:20.118298 104328 storage/store_snapshot.go:995 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] sending LEARNER snapshot 742a7312 at applied index 33 I190903 18:32:20.118534 104328 storage/store_snapshot.go:1038 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 17, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:20.128314 104542 storage/replica_raftstorage.go:793 [n2,s2,r128/3:{-}] applying LEARNER snapshot [id=742a7312 index=33] I190903 18:32:20.128466 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.133313 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.138125 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:20.143868 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.147191 104542 storage/replica_raftstorage.go:814 [n2,s2,r128/3:/Table/76/1/{4-5}] applied LEARNER snapshot [total=19ms ingestion=4@18ms id=742a7312 index=33] I190903 18:32:20.147908 104767 storage/replica_command.go:1430 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r127:/Table/76/1/{3-4} [(n3,s3):2, next=3, gen=92] I190903 18:32:20.149718 104328 storage/replica_command.go:1430 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r128:/Table/76/1/{4-5} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=95] I190903 18:32:20.153608 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.154220 104328 storage/replica_raft.go:291 [n1,s1,r128/2:/Table/76/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:20.155594 104328 storage/replica_command.go:1430 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r128:/Table/76/1/{4-5} [(n1,s1):2, (n2,s2):3, next=4, gen=96] I190903 18:32:20.159931 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.163625 104767 storage/replica_raft.go:291 [n3,s3,r127/2:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 18:32:20.165290 104767 storage/store_snapshot.go:995 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] sending LEARNER snapshot 29b5452d at applied index 32 I190903 18:32:20.165738 104767 storage/store_snapshot.go:1038 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:20.166263 104328 storage/replica_raft.go:291 [n1,s1,r128/2:/Table/76/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:20.166298 104328 storage/replica_raft.go:304 [n1,s1,r128/2:/Table/76/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:20.166414 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.167041 104887 storage/replica_raftstorage.go:793 [n1,s1,r127/3:{-}] applying LEARNER snapshot [id=29b5452d index=32] I190903 18:32:20.167228 104328 storage/replica_command.go:979 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r128 failed: storage/replica_raft.go:305: [n1,s1,r128/2:/Table/76/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:20.167258 104328 storage/queue.go:1027 [n1,replicate,s1,r128/2:/Table/76/1/{4-5}] change replicas of r128 failed: storage/replica_raft.go:305: [n1,s1,r128/2:/Table/76/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:20.168460 104887 storage/replica_raftstorage.go:814 [n1,s1,r127/3:/Table/76/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=29b5452d index=32] I190903 18:32:20.168960 104767 storage/replica_command.go:1430 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] change replicas (add [(n1,s1):3] remove []): existing descriptor r127:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=93] I190903 18:32:20.183040 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.188850 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.189509 104767 storage/replica_raft.go:291 [n3,s3,r127/2:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 18:32:20.190139 104767 storage/replica_command.go:1430 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r127:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=94] I190903 18:32:20.194282 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.200651 104767 storage/replica_raft.go:291 [n3,s3,r127/2:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 18:32:20.200693 104767 storage/replica_raft.go:304 [n3,s3,r127/2:/Table/76/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:20.200928 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.201589 104767 storage/replica_command.go:979 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:20.201628 104767 storage/queue.go:1027 [n3,replicate,s3,r127/2:/Table/76/1/{3-4}] change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:20.209064 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.213256 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:20.223866 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:20.227863 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.238579 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.247034 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.256229 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.264123 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.270317 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:20.298911 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.307159 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.315002 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:20.325548 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.353657 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.387217 105611 storage/replica_command.go:1430 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r126:/Table/76/1{-/3} [(n2,s2):1, next=2, gen=88] W190903 18:32:20.388011 105833 storage/replica_command.go:1917 [n3,s3,r127/2:/Table/76/1/{3-4}] while transferring lease: unable to find store 2 in range r127:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=94] I190903 18:32:20.388340 105833 storage/replica_command.go:1430 [n3,s3,r127/2:/Table/76/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r127:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=94] I190903 18:32:20.391020 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:20.394317 105611 storage/replica_raft.go:291 [n2,s2,r126/1:/Table/76/1{-/3}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:20.394846 105833 storage/replica_raft.go:291 [n3,s3,r127/2:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 18:32:20.394890 105833 storage/replica_raft.go:304 [n3,s3,r127/2:/Table/76/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:20.395572 105611 storage/store_snapshot.go:995 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] sending LEARNER snapshot 47434c99 at applied index 23 I190903 18:32:20.395738 105833 storage/replica_command.go:979 [n3,s3,r127/2:/Table/76/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:20.395761 105611 storage/store_snapshot.go:1038 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s W190903 18:32:20.396173 105833 storage/replica_command.go:2033 [n3,s3,r127/2:/Table/76/1/{3-4}] while removing target n3,s3: change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:20.396389 105803 storage/queue.go:1027 [n2,merge,s2,r126/1:/Table/76/1{-/3}] change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:20.396885 105826 storage/replica_raftstorage.go:793 [n1,s1,r126/2:{-}] applying LEARNER snapshot [id=47434c99 index=23] I190903 18:32:20.406344 105826 storage/replica_raftstorage.go:814 [n1,s1,r126/2:/Table/76/1{-/3}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=47434c99 index=23] I190903 18:32:20.406847 105611 storage/replica_command.go:1430 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] change replicas (add [(n1,s1):2] remove []): existing descriptor r126:/Table/76/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=89] I190903 18:32:20.414532 105611 storage/replica_raft.go:291 [n2,s2,r126/1:/Table/76/1{-/3}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:20.415136 105611 storage/replica_command.go:1430 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r126:/Table/76/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=90] I190903 18:32:20.429443 105611 storage/replica_raft.go:291 [n2,s2,r126/1:/Table/76/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:20.429482 105611 storage/replica_raft.go:304 [n2,s2,r126/1:/Table/76/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:20.430546 105611 storage/replica_command.go:979 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] could not promote [] to voter, rolling back: change replicas of r126 failed: storage/replica_raft.go:305: [n2,s2,r126/1:/Table/76/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:20.430583 105611 storage/queue.go:1027 [n2,replicate,s2,r126/1:/Table/76/1{-/3}] change replicas of r126 failed: storage/replica_raft.go:305: [n2,s2,r126/1:/Table/76/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:20.431890 105923 storage/replica_command.go:1430 [n2,replicate,s2,r128/3:/Table/76/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r128:/Table/76/1/{4-5} [(n1,s1):2, (n2,s2):3, next=4, gen=96] I190903 18:32:20.443208 105923 storage/replica_raft.go:291 [n2,s2,r128/3:/Table/76/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:20.452492 106007 storage/store.go:2593 [n1,replicaGC,s1,r128/2:/Table/76/1/{4-5}] removing replica r128/2 I190903 18:32:20.452848 106007 storage/replica_destroy.go:152 [n1,replicaGC,s1,r128/2:/Table/76/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:20.462033 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:20.613726 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 18:32:20.660508 106649 storage/queue.go:1027 [n1,replicate,s1,r127/3:/Table/76/1/{3-4}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 18:32:20.743756 106543 storage/replica_command.go:1430 [n1,replicate,s1,r127/3:/Table/76/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r127:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=94] I190903 18:32:20.749730 106543 storage/replica_raft.go:291 [n1,s1,r127/3:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 18:32:20.760436 106884 storage/store.go:2593 [n3,replicaGC,s3,r127/2:/Table/76/1/{3-4}] removing replica r127/2 I190903 18:32:20.760877 106884 storage/replica_destroy.go:152 [n3,replicaGC,s3,r127/2:/Table/76/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:20.762231 105897 storage/queue.go:1027 [n1,replicate,s1,r126/2:/Table/76/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:31] E190903 18:32:20.809210 106912 storage/queue.go:1027 [n2,merge,s2,r126/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):3 I190903 18:32:20.887031 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:20.949218 107359 storage/replica_command.go:1430 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r96:/Table/76/{1/5-2} [(n1,s1):1, next=2, gen=94] I190903 18:32:20.964414 107359 storage/replica_raft.go:291 [n1,s1,r96/1:/Table/76/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:32:20.971055 107359 storage/store_snapshot.go:995 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] sending LEARNER snapshot 9d6ecd3c at applied index 15 I190903 18:32:20.971948 107359 storage/store_snapshot.go:1038 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:20.974973 105651 storage/replica_raftstorage.go:793 [n3,s3,r96/2:{-}] applying LEARNER snapshot [id=9d6ecd3c index=15] I190903 18:32:20.976338 105651 storage/replica_raftstorage.go:814 [n3,s3,r96/2:/Table/76/{1/5-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=9d6ecd3c index=15] I190903 18:32:20.977372 107359 storage/replica_command.go:1430 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] change replicas (add [(n3,s3):2] remove []): existing descriptor r96:/Table/76/{1/5-2} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=95] I190903 18:32:20.982075 107359 storage/replica_raft.go:291 [n1,s1,r96/1:/Table/76/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:32:20.984009 107359 storage/replica_command.go:1430 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r96:/Table/76/{1/5-2} [(n1,s1):1, (n3,s3):2, next=3, gen=96] I190903 18:32:20.994836 107359 storage/replica_raft.go:291 [n1,s1,r96/1:/Table/76/{1/5-2}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:20.994879 107359 storage/replica_raft.go:304 [n1,s1,r96/1:/Table/76/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:20.995708 107359 storage/replica_command.go:979 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r96 failed: storage/replica_raft.go:305: [n1,s1,r96/1:/Table/76/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:20.995742 107359 storage/queue.go:1027 [n1,replicate,s1,r96/1:/Table/76/{1/5-2}] change replicas of r96 failed: storage/replica_raft.go:305: [n1,s1,r96/1:/Table/76/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:21.129308 107885 storage/queue.go:1027 [n3,replicate,s3,r96/2:/Table/76/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:22] E190903 18:32:21.324035 108314 storage/queue.go:1027 [n1,merge,s1,r126/2:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):3 I190903 18:32:21.377855 108242 storage/replica_command.go:1430 [n1,replicate,s1,r126/2:/Table/76/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r126:/Table/76/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=90] I190903 18:32:21.382434 108242 storage/replica_raft.go:291 [n1,s1,r126/2:/Table/76/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:21.390741 107715 storage/store.go:2593 [n2,replicaGC,s2,r126/1:/Table/76/1{-/3}] removing replica r126/1 I190903 18:32:21.391106 107715 storage/replica_destroy.go:152 [n2,replicaGC,s2,r126/1:/Table/76/1{-/3}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:21.425955 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:21.428500 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:21.435717 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:21.535268 108952 storage/replica_command.go:609 [n1,merge,s1,r126/2:/Table/76/1{-/3}] initiating a merge of r127:/Table/76/1/{3-4} [(n1,s1):3, next=4, gen=95] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:21.542910 200 storage/store.go:2593 [n1,s1,r126/2:/Table/76/1{-/3}] removing replica r127/3 I190903 18:32:21.772031 109315 storage/replica_command.go:1430 [n3,replicate,s3,r96/2:/Table/76/{1/5-2}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r96:/Table/76/{1/5-2} [(n1,s1):1, (n3,s3):2, next=3, gen=96] I190903 18:32:21.780757 109315 storage/replica_raft.go:291 [n3,s3,r96/2:/Table/76/{1/5-2}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 18:32:21.789821 109581 storage/store.go:2593 [n1,replicaGC,s1,r96/1:/Table/76/{1/5-2}] removing replica r96/1 I190903 18:32:21.790141 109581 storage/replica_destroy.go:152 [n1,replicaGC,s1,r96/1:/Table/76/{1/5-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:22.485967 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.488038 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:22.491353 96675 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 === RUN TestRepartitioning/single_col_range_partitioning_-_MAXVALUE/single_col_range_partitioning I190903 18:32:22.511182 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=76 (single col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 18:32:22.494547947 +0000 UTC I190903 18:32:22.534856 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 75, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning - MAXVALUE"]} I190903 18:32:22.537192 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 4}] I190903 18:32:22.593227 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=76 (single col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 18:32:22.593058327 +0000 UTC I190903 18:32:22.619695 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 77, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:22.629373 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 78, info: {TableName:data.public."single col range partitioning - MAXVALUE" Statement:CREATE TABLE "single col range partitioning - MAXVALUE" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p4 VALUES FROM (minvalue) TO (4), PARTITION p5 VALUES FROM (4) TO (5), PARTITION px VALUES FROM (5) TO (maxvalue)) User:root} I190903 18:32:22.634867 112959 storage/replica_command.go:284 [n2,split,s2,r129/1:/{Table/76/2-Max}] initiating a split of this range at key /Table/78 [r130] (zone config) I190903 18:32:22.650972 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:22.653681 1537 storage/queue.go:518 [n2,s2,r129/1:/Table/7{6/2-8}] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:22.655704 111794 storage/replica_command.go:284 [n2,split,s2,r130/1:/{Table/78-Max}] initiating a split of this range at key /Table/78/1 [r141] (zone config) I190903 18:32:22.688575 113393 storage/replica_command.go:284 [n2,split,s2,r141/1:/{Table/78/1-Max}] initiating a split of this range at key /Table/78/1/4 [r142] (zone config) I190903 18:32:22.692151 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p5 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:22.792509 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION px OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:22.795262 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.812221 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:22.814473 113958 storage/replica_command.go:1430 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r141:/Table/78/1{-/4} [(n2,s2):1, next=2, gen=93] I190903 18:32:22.814693 114340 storage/replica_command.go:284 [n2,split,s2,r142/1:/{Table/78/1/4-Max}] initiating a split of this range at key /Table/78/1/5 [r143] (zone config) I190903 18:32:22.832071 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:22.834639 113958 storage/replica_raft.go:291 [n2,s2,r141/1:/Table/78/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:22.835881 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.838726 113958 storage/store_snapshot.go:995 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] sending LEARNER snapshot a7c51e25 at applied index 18 I190903 18:32:22.838944 113958 storage/store_snapshot.go:1038 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:22.840103 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.843765 114429 storage/replica_raftstorage.go:793 [n1,s1,r141/2:{-}] applying LEARNER snapshot [id=a7c51e25 index=18] I190903 18:32:22.845402 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:22.856244 114429 storage/replica_raftstorage.go:814 [n1,s1,r141/2:/Table/78/1{-/4}] applied LEARNER snapshot [total=12ms ingestion=4@12ms id=a7c51e25 index=18] I190903 18:32:22.867311 113958 storage/replica_command.go:1430 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r141:/Table/78/1{-/4} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=94] I190903 18:32:22.868477 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.879582 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.884921 114223 storage/replica_command.go:284 [n2,split,s2,r143/1:/{Table/78/1/5-Max}] initiating a split of this range at key /Table/78/2 [r144] (zone config) I190903 18:32:22.885380 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:22.887819 113958 storage/replica_raft.go:291 [n2,s2,r141/1:/Table/78/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:22.888412 113958 storage/replica_command.go:1430 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r141:/Table/78/1{-/4} [(n2,s2):1, (n1,s1):2, next=3, gen=95] I190903 18:32:22.892235 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:22.901570 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.902945 113958 storage/replica_raft.go:291 [n2,s2,r141/1:/Table/78/1{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:22.902990 113958 storage/replica_raft.go:304 [n2,s2,r141/1:/Table/78/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:22.905509 113958 storage/replica_command.go:979 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] could not promote [] to voter, rolling back: change replicas of r141 failed: storage/replica_raft.go:305: [n2,s2,r141/1:/Table/78/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:22.905547 113958 storage/queue.go:1027 [n2,replicate,s2,r141/1:/Table/78/1{-/4}] change replicas of r141 failed: storage/replica_raft.go:305: [n2,s2,r141/1:/Table/78/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:22.915742 114736 storage/replica_command.go:1430 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r143:/Table/78/{1/5-2} [(n2,s2):1, next=2, gen=95] I190903 18:32:22.915805 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.926876 114736 storage/replica_raft.go:291 [n2,s2,r143/1:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:22.928123 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.931322 114736 storage/store_snapshot.go:995 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] sending LEARNER snapshot 8b650d0e at applied index 17 I190903 18:32:22.931385 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.931723 114736 storage/store_snapshot.go:1038 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:22.932959 114820 storage/replica_raftstorage.go:793 [n3,s3,r143/2:{-}] applying LEARNER snapshot [id=8b650d0e index=17] I190903 18:32:22.947309 114820 storage/replica_raftstorage.go:814 [n3,s3,r143/2:/Table/78/{1/5-2}] applied LEARNER snapshot [total=14ms ingestion=4@7ms id=8b650d0e index=17] I190903 18:32:22.954460 114736 storage/replica_command.go:1430 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] change replicas (add [(n3,s3):2] remove []): existing descriptor r143:/Table/78/{1/5-2} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=96] I190903 18:32:22.959555 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.963786 114736 storage/replica_raft.go:291 [n2,s2,r143/1:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 18:32:22.964341 114736 storage/replica_command.go:1430 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r143:/Table/78/{1/5-2} [(n2,s2):1, (n3,s3):2, next=3, gen=97] I190903 18:32:22.976054 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.986020 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.993230 114736 storage/replica_raft.go:291 [n2,s2,r143/1:/Table/78/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 18:32:22.993312 114736 storage/replica_raft.go:304 [n2,s2,r143/1:/Table/78/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:22.993638 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:22.994174 114736 storage/replica_command.go:979 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r143 failed: storage/replica_raft.go:305: [n2,s2,r143/1:/Table/78/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:22.994243 114736 storage/queue.go:1027 [n2,replicate,s2,r143/1:/Table/78/{1/5-2}] change replicas of r143 failed: storage/replica_raft.go:305: [n2,s2,r143/1:/Table/78/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:23.003082 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.010537 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.015410 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.021680 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:23.027456 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:23.041583 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.051681 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.059842 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.071786 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:23.075194 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:23.096072 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 18:32:23.128119 115559 storage/queue.go:1027 [n3,replicate,s3,r143/2:/Table/78/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:23.137020 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.208791 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:23.394774 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 18:32:23.445849 116352 storage/queue.go:1027 [n1,replicate,s1,r141/2:/Table/78/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:23.666975 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 18:32:23.670351 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 18:32:23.673082 117007 storage/replica_command.go:1430 [n3,replicate,s3,r143/2:/Table/78/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r143:/Table/78/{1/5-2} [(n2,s2):1, (n3,s3):2, next=3, gen=97] I190903 18:32:23.675053 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 18:32:23.680028 117007 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/78/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 18:32:23.688002 117096 storage/store.go:2593 [n2,replicaGC,s2,r143/1:/Table/78/{1/5-2}] removing replica r143/1 I190903 18:32:23.688370 117096 storage/replica_destroy.go:152 [n2,replicaGC,s2,r143/1:/Table/78/{1/5-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:23.718604 117177 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/78/1{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r141:/Table/78/1{-/4} [(n2,s2):1, (n1,s1):2, next=3, gen=95] I190903 18:32:23.722228 117177 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/78/1{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:23.731470 117220 storage/store.go:2593 [n2,replicaGC,s2,r141/1:/Table/78/1{-/4}] removing replica r141/1 I190903 18:32:23.731873 117220 storage/replica_destroy.go:152 [n2,replicaGC,s2,r141/1:/Table/78/1{-/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:23.735914 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=78 (single col range partitioning) version=2 mtime=2019-09-03 18:32:23.734251508 +0000 UTC I190903 18:32:23.752175 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 1}] I190903 18:32:23.758537 117258 sql/lease.go:923 new lease: 78("single col range partitioning") ver=2:1567535839.268466597,0, refcount=0 I190903 18:32:23.805600 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=78 (single col range partitioning) version=3 mtime=2019-09-03 18:32:23.805374498 +0000 UTC I190903 18:32:23.812726 117716 sql/lease.go:923 new lease: 78("single col range partitioning") ver=3:1567535840.815749045,0, refcount=0 I190903 18:32:23.823152 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=78 (single col range partitioning) version=4 mtime=2019-09-03 18:32:23.819324609 +0000 UTC I190903 18:32:23.831085 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 78, info: {TableName:data.public."single col range partitioning" Statement:ALTER TABLE "single col range partitioning" PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:23.833202 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 3}] I190903 18:32:23.834715 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:23.836784 118079 storage/replica_command.go:1430 [n2,s2,r144/1:/{Table/78/2-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r144:/{Table/78/2-Max} [(n2,s2):1, next=2, gen=95] I190903 18:32:23.837686 118075 sql/lease.go:923 new lease: 78("single col range partitioning") ver=4:1567535856.187559785,0, refcount=0 I190903 18:32:23.840151 118269 storage/replica_command.go:1430 [n3,s3,r143/2:/Table/78/{1/5-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r143:/Table/78/{1/5-2} [(n3,s3):2, next=3, gen=98] I190903 18:32:23.850061 118269 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 18:32:23.851285 118269 storage/store_snapshot.go:995 [n3,s3,r143/2:/Table/78/{1/5-2}] sending LEARNER snapshot 0112ee87 at applied index 32 I190903 18:32:23.851590 118269 storage/store_snapshot.go:1038 [n3,s3,r143/2:/Table/78/{1/5-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:23.855366 118484 storage/replica_raftstorage.go:793 [n2,s2,r143/3:{-}] applying LEARNER snapshot [id=0112ee87 index=32] I190903 18:32:23.858653 118079 storage/replica_raft.go:291 [n2,s2,r144/1:/{Table/78/2-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:23.859841 118079 storage/store_snapshot.go:995 [n2,s2,r144/1:/{Table/78/2-Max}] sending LEARNER snapshot ffbcebcd at applied index 18 I190903 18:32:23.859989 118079 storage/store_snapshot.go:1038 [n2,s2,r144/1:/{Table/78/2-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:23.861123 118516 storage/replica_raftstorage.go:793 [n3,s3,r144/2:{-}] applying LEARNER snapshot [id=ffbcebcd index=18] I190903 18:32:23.862394 118516 storage/replica_raftstorage.go:814 [n3,s3,r144/2:/{Table/78/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ffbcebcd index=18] I190903 18:32:23.863670 118079 storage/replica_command.go:1430 [n2,s2,r144/1:/{Table/78/2-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r144:/{Table/78/2-Max} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=96] I190903 18:32:23.865615 118484 storage/replica_raftstorage.go:814 [n2,s2,r143/3:/Table/78/{1/5-2}] applied LEARNER snapshot [total=10ms ingestion=4@6ms id=0112ee87 index=32] I190903 18:32:23.866231 118269 storage/replica_command.go:1430 [n3,s3,r143/2:/Table/78/{1/5-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r143:/Table/78/{1/5-2} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=99] I190903 18:32:23.881975 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 78, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:23.882704 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:23.883234 118269 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 I190903 18:32:23.885952 118810 storage/replica_command.go:284 [n1,split,s1,r141/2:/Table/78/1{-/4}] initiating a split of this range at key /Table/78/1/3 [r97] (zone config) E190903 18:32:23.887162 118603 storage/queue.go:1027 [n2,replicate,s2,r143/3:/Table/78/{1/5-2}] no removable replicas from range that needs a removal: [2:0, 3*:36] I190903 18:32:23.887509 118538 storage/replica_command.go:1430 [n2,s2,r143/3:/Table/78/{1/5-2}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r143:/Table/78/{1/5-2} [(n3,s3):2, (n2,s2):3, next=4, gen=100] I190903 18:32:23.887589 118079 storage/replica_raft.go:291 [n2,s2,r144/1:/{Table/78/2-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 18:32:23.891199 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} E190903 18:32:23.910334 119235 storage/queue.go:1027 [n3,replicate,s3,r144/2:/{Table/78/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:23.911345 118538 storage/replica_raft.go:291 [n2,s2,r143/3:/Table/78/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 I190903 18:32:23.916000 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:23.916271 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.916725 118324 storage/replica_command.go:609 [n2,merge,s2,r142/1:/Table/78/1/{4-5}] initiating a merge of r143:/Table/78/{1/5-2} [(n2,s2):3, next=4, gen=101] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:23.936088 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.940243 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.942889 119319 storage/store.go:2593 [n3,replicaGC,s3,r143/2:/Table/78/{1/5-2}] removing replica r143/2 I190903 18:32:23.944353 119319 storage/replica_destroy.go:152 [n3,replicaGC,s3,r143/2:/Table/78/{1/5-2}] removed 12 (0+12) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:23.945464 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.947913 1551 storage/store.go:2593 [n2,s2,r142/1:/Table/78/1/{4-5}] removing replica r143/3 I190903 18:32:23.952951 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.957915 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.963023 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.968598 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:23.972214 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.974426 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:23.976336 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:24.002417 119865 storage/replica_command.go:1430 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r97:/Table/78/1/{3-4} [(n1,s1):1, next=2, gen=97] I190903 18:32:24.011885 119266 storage/replica_command.go:1430 [n3,s3,r144/2:/{Table/78/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r144:/{Table/78/2-Max} [(n2,s2):1, (n3,s3):2, next=3, gen=97] I190903 18:32:24.023553 119865 storage/replica_raft.go:291 [n1,s1,r97/1:/Table/78/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 18:32:24.023598 119266 storage/replica_raft.go:291 [n3,s3,r144/2:/{Table/78/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 18:32:24.024625 119865 storage/store_snapshot.go:995 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] sending LEARNER snapshot 9f3a9477 at applied index 15 I190903 18:32:24.024810 119865 storage/store_snapshot.go:1038 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:24.026134 119901 storage/replica_raftstorage.go:793 [n3,s3,r97/2:{-}] applying LEARNER snapshot [id=9f3a9477 index=15] I190903 18:32:24.027550 119901 storage/replica_raftstorage.go:814 [n3,s3,r97/2:/Table/78/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=9f3a9477 index=15] I190903 18:32:24.027935 119865 storage/replica_command.go:1430 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r97:/Table/78/1/{3-4} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=98] I190903 18:32:24.037286 119729 storage/store.go:2593 [n2,replicaGC,s2,r144/1:/{Table/78/2-Max}] removing replica r144/1 I190903 18:32:24.037701 119729 storage/replica_destroy.go:152 [n2,replicaGC,s2,r144/1:/{Table/78/2-Max}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:24.038612 119865 storage/replica_raft.go:291 [n1,s1,r97/1:/Table/78/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 18:32:24.041379 119865 storage/replica_command.go:1430 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r97:/Table/78/1/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=99] I190903 18:32:24.048861 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.050967 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.052649 119865 storage/replica_raft.go:291 [n1,s1,r97/1:/Table/78/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 18:32:24.052771 119865 storage/replica_raft.go:304 [n1,s1,r97/1:/Table/78/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:24.052869 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.053907 119865 storage/replica_command.go:979 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r97 failed: storage/replica_raft.go:305: [n1,s1,r97/1:/Table/78/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:24.055045 119865 storage/queue.go:1027 [n1,replicate,s1,r97/1:/Table/78/1/{3-4}] change replicas of r97 failed: storage/replica_raft.go:305: [n1,s1,r97/1:/Table/78/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:24.058113 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.078093 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.083640 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.103334 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:24.109346 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.111945 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.118572 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.122405 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:24.129174 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.146155 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.157812 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.177316 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.211367 119567 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r141:/Table/78/1{-/3} [(n1,s1):2, next=3, gen=97] I190903 18:32:24.230602 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.240851 120475 storage/replica_command.go:1430 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r142:/Table/78/{1/4-2} [(n2,s2):1, next=2, gen=102] I190903 18:32:24.255101 119567 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/78/1{-/3}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:24.256810 119567 storage/store_snapshot.go:995 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] sending LEARNER snapshot ce578178 at applied index 36 I190903 18:32:24.257008 119567 storage/store_snapshot.go:1038 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:24.258449 120629 storage/replica_raftstorage.go:793 [n2,s2,r141/3:{-}] applying LEARNER snapshot [id=ce578178 index=36] I190903 18:32:24.260369 120629 storage/replica_raftstorage.go:814 [n2,s2,r141/3:/Table/78/1{-/3}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=ce578178 index=36] I190903 18:32:24.269741 119567 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] change replicas (add [(n2,s2):3] remove []): existing descriptor r141:/Table/78/1{-/3} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=98] I190903 18:32:24.274021 119567 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/78/1{-/3}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:24.274624 119567 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r141:/Table/78/1{-/3} [(n1,s1):2, (n2,s2):3, next=4, gen=99] I190903 18:32:24.284496 120475 storage/replica_raft.go:291 [n2,s2,r142/1:/Table/78/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:24.285191 119567 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/78/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:24.285231 119567 storage/replica_raft.go:304 [n1,s1,r141/2:/Table/78/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:24.287327 119567 storage/replica_command.go:979 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] could not promote [] to voter, rolling back: change replicas of r141 failed: storage/replica_raft.go:305: [n1,s1,r141/2:/Table/78/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:24.287369 119567 storage/queue.go:1027 [n1,replicate,s1,r141/2:/Table/78/1{-/3}] change replicas of r141 failed: storage/replica_raft.go:305: [n1,s1,r141/2:/Table/78/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:24.291511 120475 storage/store_snapshot.go:995 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] sending LEARNER snapshot e65fe07b at applied index 26 I190903 18:32:24.291756 120475 storage/store_snapshot.go:1038 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 24, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:24.293279 120549 storage/replica_raftstorage.go:793 [n1,s1,r142/2:{-}] applying LEARNER snapshot [id=e65fe07b index=26] I190903 18:32:24.307701 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:24.308075 120549 storage/replica_raftstorage.go:814 [n1,s1,r142/2:/Table/78/{1/4-2}] applied LEARNER snapshot [total=15ms ingestion=4@14ms id=e65fe07b index=26] I190903 18:32:24.308683 120475 storage/replica_command.go:1430 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r142:/Table/78/{1/4-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=103] I190903 18:32:24.316372 120475 storage/replica_raft.go:291 [n2,s2,r142/1:/Table/78/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:24.317023 120475 storage/replica_command.go:1430 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r142:/Table/78/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=104] I190903 18:32:24.342476 120475 storage/replica_raft.go:291 [n2,s2,r142/1:/Table/78/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:24.342532 120475 storage/replica_raft.go:304 [n2,s2,r142/1:/Table/78/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:24.343514 120475 storage/replica_command.go:979 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r142 failed: storage/replica_raft.go:305: [n2,s2,r142/1:/Table/78/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:24.343555 120475 storage/queue.go:1027 [n2,replicate,s2,r142/1:/Table/78/{1/4-2}] change replicas of r142 failed: storage/replica_raft.go:305: [n2,s2,r142/1:/Table/78/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:24.445081 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:24.717054 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 18:32:24.767654 121975 storage/queue.go:1027 [n2,replicate,s2,r141/3:/Table/78/1{-/3}] no removable replicas from range that needs a removal: [2:0, 3*:44] E190903 18:32:24.787235 122037 storage/queue.go:1027 [n1,replicate,s1,r142/2:/Table/78/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:34] E190903 18:32:24.881823 122293 storage/queue.go:1027 [n3,replicate,s3,r97/2:/Table/78/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:24.975694 122353 storage/replica_command.go:1430 [n1,replicate,s1,r142/2:/Table/78/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r142:/Table/78/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=104] I190903 18:32:24.979508 122353 storage/replica_raft.go:291 [n1,s1,r142/2:/Table/78/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:24.988121 122564 storage/store.go:2593 [n2,replicaGC,s2,r142/1:/Table/78/{1/4-2}] removing replica r142/1 I190903 18:32:24.988580 122564 storage/replica_destroy.go:152 [n2,replicaGC,s2,r142/1:/Table/78/{1/4-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:25.022190 122473 storage/replica_command.go:1430 [n2,replicate,s2,r141/3:/Table/78/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r141:/Table/78/1{-/3} [(n1,s1):2, (n2,s2):3, next=4, gen=99] I190903 18:32:25.031404 122473 storage/replica_raft.go:291 [n2,s2,r141/3:/Table/78/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:25.094269 122715 storage/store.go:2593 [n1,replicaGC,s1,r141/2:/Table/78/1{-/3}] removing replica r141/2 I190903 18:32:25.094742 122715 storage/replica_destroy.go:152 [n1,replicaGC,s1,r141/2:/Table/78/1{-/3}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:25.136028 122963 storage/replica_command.go:1430 [n3,replicate,s3,r97/2:/Table/78/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r97:/Table/78/1/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=99] I190903 18:32:25.146231 122963 storage/replica_raft.go:291 [n3,s3,r97/2:/Table/78/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 18:32:25.155390 123018 storage/store.go:2593 [n1,replicaGC,s1,r97/1:/Table/78/1/{3-4}] removing replica r97/1 I190903 18:32:25.155716 123018 storage/replica_destroy.go:152 [n1,replicaGC,s1,r97/1:/Table/78/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:25.256168 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:25.259732 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:25.262818 111440 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 === RUN TestRepartitioning/multi_col_range_partitioning/multi_col_range_partitioning_-_MAXVALUE I190903 18:32:25.291567 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=78 (single col range partitioning) version=5 mtime=2019-09-03 18:32:25.264773131 +0000 UTC I190903 18:32:25.322960 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 77, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning"]} I190903 18:32:25.325208 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 4}] I190903 18:32:25.326939 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:25.328293 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:25.384120 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=78 (single col range partitioning) version=6 mtime=2019-09-03 18:32:25.383956357 +0000 UTC I190903 18:32:25.411638 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 79, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:25.419653 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 80, info: {TableName:data.public."multi col range partitioning" Statement:CREATE TABLE "multi col range partitioning" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p56 VALUES FROM (3, 4) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root} I190903 18:32:25.424579 124817 storage/replica_command.go:284 [n3,split,s3,r144/2:/{Table/78/2-Max}] initiating a split of this range at key /Table/80 [r67] (zone config) I190903 18:32:25.455606 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:25.458309 125211 storage/replica_command.go:284 [n3,split,s3,r67/1:/{Table/80-Max}] initiating a split of this range at key /Table/80/1 [r68] (zone config) I190903 18:32:25.468954 125218 storage/replica_command.go:284 [n3,split,s3,r68/1:/{Table/80/1-Max}] initiating a split of this range at key /Table/80/2 [r69] (zone config) I190903 18:32:25.470606 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:25.520433 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:25.588623 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:25.593689 126075 storage/replica_command.go:284 [n3,split,s3,r68/1:/Table/80/{1-2}] initiating a split of this range at key /Table/80/1/3/4 [r70] (zone config) I190903 18:32:25.628818 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:25.629126 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.645922 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.652173 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.652381 126736 storage/replica_command.go:284 [n3,split,s3,r70/1:/Table/80/{1/3/4-2}] initiating a split of this range at key /Table/80/1/5/6 [r71] (zone config) I190903 18:32:25.653048 126429 storage/replica_command.go:1430 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r68:/Table/80/1{-/3/4} [(n3,s3):1, next=2, gen=102] I190903 18:32:25.657140 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.664643 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.668274 126429 storage/replica_raft.go:291 [n3,s3,r68/1:/Table/80/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:25.670901 126429 storage/store_snapshot.go:995 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] sending LEARNER snapshot 70a9c62a at applied index 21 I190903 18:32:25.671726 126429 storage/store_snapshot.go:1038 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:25.672369 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.673248 126922 storage/replica_command.go:284 [n3,split,s3,r71/1:/Table/80/{1/5/6-2}] initiating a split of this range at key /Table/80/1/5/7 [r72] (zone config) I190903 18:32:25.673658 126813 storage/replica_raftstorage.go:793 [n2,s2,r68/2:{-}] applying LEARNER snapshot [id=70a9c62a index=21] I190903 18:32:25.677877 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.678914 126813 storage/replica_raftstorage.go:814 [n2,s2,r68/2:/Table/80/1{-/3/4}] applied LEARNER snapshot [total=5ms ingestion=4@2ms id=70a9c62a index=21] I190903 18:32:25.684515 126429 storage/replica_command.go:1430 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r68:/Table/80/1{-/3/4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=103] I190903 18:32:25.686287 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.694972 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.705757 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:25.711803 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.716238 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.723353 126429 storage/replica_raft.go:291 [n3,s3,r68/1:/Table/80/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:25.731869 126429 storage/replica_command.go:1430 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r68:/Table/80/1{-/3/4} [(n3,s3):1, (n2,s2):2, next=3, gen=104] I190903 18:32:25.744195 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.753917 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:25.755715 126429 storage/replica_raft.go:291 [n3,s3,r68/1:/Table/80/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:25.755753 126429 storage/replica_raft.go:304 [n3,s3,r68/1:/Table/80/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:25.760694 126429 storage/replica_command.go:979 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r68 failed: storage/replica_raft.go:305: [n3,s3,r68/1:/Table/80/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:25.760729 126429 storage/queue.go:1027 [n3,replicate,s3,r68/1:/Table/80/1{-/3/4}] change replicas of r68 failed: storage/replica_raft.go:305: [n3,s3,r68/1:/Table/80/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:25.761155 127168 storage/replica_command.go:1430 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r71:/Table/80/1/5/{6-7} [(n3,s3):1, next=2, gen=104] I190903 18:32:25.762649 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.768151 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.774962 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:25.804551 127168 storage/replica_raft.go:291 [n3,s3,r71/1:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:25.806844 127168 storage/store_snapshot.go:995 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] sending LEARNER snapshot a8039590 at applied index 18 I190903 18:32:25.807618 127168 storage/store_snapshot.go:1038 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:25.808842 127394 storage/replica_raftstorage.go:793 [n1,s1,r71/2:{-}] applying LEARNER snapshot [id=a8039590 index=18] I190903 18:32:25.814144 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.818361 127394 storage/replica_raftstorage.go:814 [n1,s1,r71/2:/Table/80/1/5/{6-7}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=a8039590 index=18] I190903 18:32:25.824698 127168 storage/replica_command.go:1430 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] change replicas (add [(n1,s1):2] remove []): existing descriptor r71:/Table/80/1/5/{6-7} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105] I190903 18:32:25.826286 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:25.843359 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.846839 127168 storage/replica_raft.go:291 [n3,s3,r71/1:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:25.847397 127168 storage/replica_command.go:1430 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r71:/Table/80/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:25.856779 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:25.868465 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:25.869382 127168 storage/replica_raft.go:291 [n3,s3,r71/1:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:25.869430 127168 storage/replica_raft.go:304 [n3,s3,r71/1:/Table/80/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:25.870250 127168 storage/replica_command.go:979 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r71 failed: storage/replica_raft.go:305: [n3,s3,r71/1:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:25.870291 127168 storage/queue.go:1027 [n3,replicate,s3,r71/1:/Table/80/1/5/{6-7}] change replicas of r71 failed: storage/replica_raft.go:305: [n3,s3,r71/1:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:25.870762 127627 storage/replica_command.go:1430 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r72:/Table/80/{1/5/7-2} [(n3,s3):1, next=2, gen=104] I190903 18:32:25.878091 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:25.900525 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.901507 127627 storage/replica_raft.go:291 [n3,s3,r72/1:/Table/80/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:25.906466 127627 storage/store_snapshot.go:995 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] sending LEARNER snapshot 5c885702 at applied index 15 I190903 18:32:25.906651 127627 storage/store_snapshot.go:1038 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:25.908230 127612 storage/replica_raftstorage.go:793 [n1,s1,r72/2:{-}] applying LEARNER snapshot [id=5c885702 index=15] I190903 18:32:25.914865 127612 storage/replica_raftstorage.go:814 [n1,s1,r72/2:/Table/80/{1/5/7-2}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=5c885702 index=15] I190903 18:32:25.916955 127627 storage/replica_command.go:1430 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r72:/Table/80/{1/5/7-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105] I190903 18:32:25.926635 127627 storage/replica_raft.go:291 [n3,s3,r72/1:/Table/80/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:25.927454 127627 storage/replica_command.go:1430 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r72:/Table/80/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:25.934061 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:25.939322 127627 storage/replica_raft.go:291 [n3,s3,r72/1:/Table/80/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:25.939369 127627 storage/replica_raft.go:304 [n3,s3,r72/1:/Table/80/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:25.940859 127627 storage/replica_command.go:979 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r72 failed: storage/replica_raft.go:305: [n3,s3,r72/1:/Table/80/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:25.940906 127627 storage/queue.go:1027 [n3,replicate,s3,r72/1:/Table/80/{1/5/7-2}] change replicas of r72 failed: storage/replica_raft.go:305: [n3,s3,r72/1:/Table/80/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:25.944281 127891 storage/queue.go:1027 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:25.958716 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:26.009503 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 18:32:26.057262 128167 storage/queue.go:1027 [n1,replicate,s1,r72/2:/Table/80/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:21] I190903 18:32:26.079948 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:26.089287 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:26.159272 128616 storage/replica_command.go:1430 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r71:/Table/80/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:26.160602 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:26.163578 128616 storage/replica_raft.go:291 [n1,s1,r71/2:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:26.175572 128620 storage/store.go:2593 [n3,replicaGC,s3,r71/1:/Table/80/1/5/{6-7}] removing replica r71/1 I190903 18:32:26.175946 128620 storage/replica_destroy.go:152 [n3,replicaGC,s3,r71/1:/Table/80/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:26.220973 127442 storage/queue.go:1027 [n2,replicate,s2,r68/2:/Table/80/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 18:32:26.305274 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:26.307552 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:26.427533 129358 storage/replica_command.go:1430 [n1,replicate,s1,r72/2:/Table/80/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r72:/Table/80/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:26.430838 230 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:26.431499 129358 storage/replica_raft.go:291 [n1,s1,r72/2:/Table/80/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:26.441252 129362 storage/store.go:2593 [n3,replicaGC,s3,r72/1:/Table/80/{1/5/7-2}] removing replica r72/1 I190903 18:32:26.441616 129362 storage/replica_destroy.go:152 [n3,replicaGC,s3,r72/1:/Table/80/{1/5/7-2}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:26.636029 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:26.640932 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:26.644485 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:26.646407 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:26.650400 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=80 (multi col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 18:32:26.648266687 +0000 UTC I190903 18:32:26.657117 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 1}] I190903 18:32:26.663109 129921 sql/lease.go:923 new lease: 80("multi col range partitioning - MAXVALUE") ver=2:1567535850.371200693,0, refcount=0 I190903 18:32:26.682171 130330 storage/replica_command.go:1430 [n2,replicate,s2,r68/2:/Table/80/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r68:/Table/80/1{-/3/4} [(n3,s3):1, (n2,s2):2, next=3, gen=104] I190903 18:32:26.691837 130330 storage/replica_raft.go:291 [n2,s2,r68/2:/Table/80/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:26.700276 130011 storage/store.go:2593 [n3,replicaGC,s3,r68/1:/Table/80/1{-/3/4}] removing replica r68/1 I190903 18:32:26.700696 130011 storage/replica_destroy.go:152 [n3,replicaGC,s3,r68/1:/Table/80/1{-/3/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:26.710122 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=80 (multi col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 18:32:26.709931194 +0000 UTC I190903 18:32:26.716148 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 2}] I190903 18:32:26.720401 130452 sql/lease.go:923 new lease: 80("multi col range partitioning - MAXVALUE") ver=3:1567535850.372200693,0, refcount=0 I190903 18:32:26.739771 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=80 (multi col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 18:32:26.737499264 +0000 UTC I190903 18:32:26.748929 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 80, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:ALTER TABLE "multi col range partitioning - MAXVALUE" PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:26.751453 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 3}] I190903 18:32:26.751575 130306 sql/lease.go:923 new lease: 80("multi col range partitioning - MAXVALUE") ver=4:1567535850.373200693,0, refcount=0 I190903 18:32:26.797054 281 server/status/runtime.go:498 [n1] runtime stats: 252 MiB RSS, 956 goroutines, 64 MiB/35 MiB/138 MiB GO alloc/idle/total, 53 MiB/105 MiB CGO alloc/total, 7512.9 CGO/sec, 73.1/7.7 %(u/s)time, 0.4 %gc (12x), 105 MiB/104 MiB (r/w)net I190903 18:32:26.807929 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:26.810745 131495 storage/replica_command.go:284 [n3,split,s3,r70/1:/Table/80/1/{3/4-5/6}] initiating a split of this range at key /Table/80/1/4 [r73] (zone config) I190903 18:32:26.817433 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:26.819044 131263 storage/replica_command.go:284 [n2,split,s2,r68/2:/Table/80/1{-/3/4}] initiating a split of this range at key /Table/80/1/3 [r145] (zone config) I190903 18:32:26.823801 131502 storage/replica_command.go:1430 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r70:/Table/80/1/{3/4-4} [(n3,s3):1, next=2, gen=104] I190903 18:32:26.831903 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:26.833667 131502 storage/replica_raft.go:291 [n3,s3,r70/1:/Table/80/1/{3/4-4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:26.835219 131502 storage/store_snapshot.go:995 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] sending LEARNER snapshot 2eb0878b at applied index 25 I190903 18:32:26.835388 131502 storage/store_snapshot.go:1038 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:26.836790 131907 storage/replica_raftstorage.go:793 [n1,s1,r70/2:{-}] applying LEARNER snapshot [id=2eb0878b index=25] I190903 18:32:26.838078 131907 storage/replica_raftstorage.go:814 [n1,s1,r70/2:/Table/80/1/{3/4-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=2eb0878b index=25] I190903 18:32:26.839182 131502 storage/replica_command.go:1430 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105] I190903 18:32:26.844323 131979 storage/replica_command.go:1430 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r145:/Table/80/1/3{-/4} [(n2,s2):1, next=2, gen=106] I190903 18:32:26.846399 131502 storage/replica_raft.go:291 [n3,s3,r70/1:/Table/80/1/{3/4-4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:26.846949 131502 storage/replica_command.go:1430 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:26.855363 132001 storage/replica_command.go:979 [n3,s3,r70/1:/Table/80/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r70 failed: descriptor changed: [expected] r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105] != [actual] r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=106] E190903 18:32:26.861631 132322 storage/queue.go:1027 [n2,merge,s2,r145/1:/Table/80/1/3{-/4}] removing learners from r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105]: change replicas of r70 failed: descriptor changed: [expected] r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105] != [actual] r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:26.869343 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:26.873573 131979 storage/replica_raft.go:291 [n2,s2,r145/1:/Table/80/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:26.875144 131502 storage/replica_raft.go:291 [n3,s3,r70/1:/Table/80/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:26.875187 131502 storage/replica_raft.go:304 [n3,s3,r70/1:/Table/80/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:26.875762 131979 storage/store_snapshot.go:995 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] sending LEARNER snapshot 1e22e52b at applied index 14 I190903 18:32:26.875912 131979 storage/store_snapshot.go:1038 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:26.877216 132415 storage/replica_raftstorage.go:793 [n3,s3,r145/2:{-}] applying LEARNER snapshot [id=1e22e52b index=14] I190903 18:32:26.878082 131502 storage/replica_command.go:979 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r70 failed: storage/replica_raft.go:305: [n3,s3,r70/1:/Table/80/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:26.878121 131502 storage/queue.go:1027 [n3,replicate,s3,r70/1:/Table/80/1/{3/4-4}] change replicas of r70 failed: storage/replica_raft.go:305: [n3,s3,r70/1:/Table/80/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:26.878301 132473 storage/replica_command.go:1430 [n2,s2,r145/1:/Table/80/1/3{-/4}] change replicas (add [] remove [(n3,s3):2LEARNER]): existing descriptor r145:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=107] I190903 18:32:26.878499 132415 storage/replica_raftstorage.go:814 [n3,s3,r145/2:/Table/80/1/3{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=1e22e52b index=14] I190903 18:32:26.881063 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:26.881180 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.887327 132473 storage/replica_raft.go:291 [n2,s2,r145/1:/Table/80/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1] next=3 I190903 18:32:26.887896 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 W190903 18:32:26.889764 132994 storage/replica_command.go:1917 [n3,s3,r70/1:/Table/80/1/{3/4-4}] while transferring lease: unable to find store 2 in range r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:26.890242 132994 storage/replica_command.go:1430 [n3,s3,r70/1:/Table/80/1/{3/4-4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r70:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:26.893263 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:26.894160 131979 storage/replica_command.go:979 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] could not promote [n3,s3] to voter, rolling back: change replicas of r145 failed: descriptor changed: [expected] r145:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=107] != [actual] r145:/Table/80/1/3{-/4} [(n2,s2):1, next=3, gen=108] I190903 18:32:26.895285 133114 storage/replica_command.go:1430 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r145:/Table/80/1/3{-/4} [(n2,s2):1, next=3, gen=108] I190903 18:32:26.895732 132442 storage/store.go:2593 [n3,replicaGC,s3,r145/2:/Table/80/1/3{-/4}] removing replica r145/2 I190903 18:32:26.896022 132442 storage/replica_destroy.go:152 [n3,replicaGC,s3,r145/2:/Table/80/1/3{-/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:26.898664 132994 storage/replica_raft.go:291 [n3,s3,r70/1:/Table/80/1/{3/4-4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 E190903 18:32:26.899814 132473 storage/queue.go:1027 [n2,merge,s2,r145/1:/Table/80/1/3{-/4}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 18:32:26.899908 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 18:32:26.901914 133130 storage/queue.go:1027 [n2,merge,s2,r145/1:/Table/80/1/3{-/4}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 18:32:26.902658 133114 storage/replica_raft.go:291 [n2,s2,r145/1:/Table/80/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):1 (n3,s3):3LEARNER] next=4 I190903 18:32:26.902930 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.903513 133114 storage/store_snapshot.go:995 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] sending LEARNER snapshot 79875324 at applied index 18 I190903 18:32:26.903633 133114 storage/store_snapshot.go:1038 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:26.904929 133319 storage/replica_raftstorage.go:793 [n3,s3,r145/3:{-}] applying LEARNER snapshot [id=79875324 index=18] I190903 18:32:26.907567 133228 storage/store.go:2593 [n1,replicaGC,s1,r70/2:/Table/80/1/{3/4-4}] removing replica r70/2 I190903 18:32:26.907904 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.908954 133228 storage/replica_destroy.go:152 [n1,replicaGC,s1,r70/2:/Table/80/1/{3/4-4}] removed 9 (0+9) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:26.912724 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.916380 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:26.920238 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.929789 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.938923 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.942259 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.947326 133319 storage/replica_raftstorage.go:814 [n3,s3,r145/3:/Table/80/1/3{-/4}] applied LEARNER snapshot [total=42ms ingestion=4@42ms id=79875324 index=18] I190903 18:32:26.947890 133114 storage/replica_command.go:1430 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r145:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):3LEARNER, next=4, gen=109] I190903 18:32:26.948761 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:26.958701 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.959290 133114 storage/replica_raft.go:291 [n2,s2,r145/1:/Table/80/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):1 (n3,s3):3] next=4 I190903 18:32:26.962343 133114 storage/replica_command.go:1430 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r145:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):3, next=4, gen=110] I190903 18:32:26.969189 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.975505 133114 storage/replica_raft.go:291 [n2,s2,r145/1:/Table/80/1/3{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 E190903 18:32:26.975541 133114 storage/replica_raft.go:304 [n2,s2,r145/1:/Table/80/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:26.977900 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.978532 133114 storage/replica_command.go:979 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r145 failed: storage/replica_raft.go:305: [n2,s2,r145/1:/Table/80/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:26.978568 133114 storage/queue.go:1027 [n2,replicate,s2,r145/1:/Table/80/1/3{-/4}] change replicas of r145 failed: storage/replica_raft.go:305: [n2,s2,r145/1:/Table/80/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:26.980825 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.987317 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:26.991779 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:26.997883 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:27.000622 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.007805 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.023011 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.031820 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.037953 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:27.040851 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:27.048753 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.054203 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.059274 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.067340 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.072571 134132 storage/replica_command.go:1430 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r71:/Table/80/1/5/{6-7} [(n1,s1):2, next=3, gen=107] I190903 18:32:27.073403 134082 storage/replica_command.go:1430 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r73:/Table/80/1/{4-5/6} [(n3,s3):1, next=2, gen=104] I190903 18:32:27.076795 134132 storage/replica_raft.go:291 [n1,s1,r71/2:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:27.079121 134132 storage/store_snapshot.go:995 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] sending LEARNER snapshot 504c7883 at applied index 34 I190903 18:32:27.079280 134132 storage/store_snapshot.go:1038 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:27.080910 134167 storage/replica_raftstorage.go:793 [n2,s2,r71/3:{-}] applying LEARNER snapshot [id=504c7883 index=34] I190903 18:32:27.082207 134167 storage/replica_raftstorage.go:814 [n2,s2,r71/3:/Table/80/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=504c7883 index=34] I190903 18:32:27.082947 134132 storage/replica_command.go:1430 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r71:/Table/80/1/5/{6-7} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=108] I190903 18:32:27.089077 134082 storage/replica_raft.go:291 [n3,s3,r73/1:/Table/80/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:27.089911 134132 storage/replica_raft.go:291 [n1,s1,r71/2:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:27.090325 134082 storage/store_snapshot.go:995 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] sending LEARNER snapshot 6dd4a7ad at applied index 18 I190903 18:32:27.090554 134082 storage/store_snapshot.go:1038 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:27.090594 134132 storage/replica_command.go:1430 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r71:/Table/80/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=109] I190903 18:32:27.091547 134137 storage/replica_raftstorage.go:793 [n1,s1,r73/2:{-}] applying LEARNER snapshot [id=6dd4a7ad index=18] I190903 18:32:27.093631 134137 storage/replica_raftstorage.go:814 [n1,s1,r73/2:/Table/80/1/{4-5/6}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=6dd4a7ad index=18] I190903 18:32:27.095608 134082 storage/replica_command.go:1430 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r73:/Table/80/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=105] I190903 18:32:27.100979 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.103815 134132 storage/replica_raft.go:291 [n1,s1,r71/2:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:27.103859 134132 storage/replica_raft.go:304 [n1,s1,r71/2:/Table/80/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:27.105880 134132 storage/replica_command.go:979 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r71 failed: storage/replica_raft.go:305: [n1,s1,r71/2:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:27.105921 134132 storage/queue.go:1027 [n1,replicate,s1,r71/2:/Table/80/1/5/{6-7}] change replicas of r71 failed: storage/replica_raft.go:305: [n1,s1,r71/2:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:27.106898 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.119500 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.151357 134082 storage/replica_raft.go:291 [n3,s3,r73/1:/Table/80/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:27.152098 134082 storage/replica_command.go:1430 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r73:/Table/80/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:27.160556 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.191528 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:27.194488 134082 storage/replica_raft.go:291 [n3,s3,r73/1:/Table/80/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:27.194614 134082 storage/replica_raft.go:304 [n3,s3,r73/1:/Table/80/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:27.195872 134082 storage/replica_command.go:979 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r73 failed: storage/replica_raft.go:305: [n3,s3,r73/1:/Table/80/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:27.195930 134082 storage/queue.go:1027 [n3,replicate,s3,r73/1:/Table/80/1/{4-5/6}] change replicas of r73 failed: storage/replica_raft.go:305: [n3,s3,r73/1:/Table/80/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:27.202252 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.224462 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.242939 1651 server/status/runtime.go:498 [n2] runtime stats: 255 MiB RSS, 973 goroutines, 80 MiB/27 MiB/138 MiB GO alloc/idle/total, 53 MiB/107 MiB CGO alloc/total, 7998.5 CGO/sec, 78.8/8.4 %(u/s)time, 0.4 %gc (13x), 104 MiB/104 MiB (r/w)net I190903 18:32:27.243712 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.270849 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.289471 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.318866 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.325775 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.368179 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.375028 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.450270 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 18:32:27.450991 135347 storage/queue.go:1027 [n2,merge,s2,r145/1:/Table/80/1/3{-/4}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 E190903 18:32:27.451837 135325 storage/queue.go:1027 [n3,replicate,s3,r145/3:/Table/80/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 3*:25] I190903 18:32:27.456631 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.498200 2361 server/status/runtime.go:498 [n3] runtime stats: 255 MiB RSS, 981 goroutines, 65 MiB/28 MiB/138 MiB GO alloc/idle/total, 54 MiB/107 MiB CGO alloc/total, 7598.2 CGO/sec, 75.2/8.1 %(u/s)time, 0.4 %gc (12x), 105 MiB/104 MiB (r/w)net I190903 18:32:27.525434 135501 storage/replica_command.go:1430 [n3,replicate,s3,r145/3:/Table/80/1/3{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r145:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):3, next=4, gen=110] I190903 18:32:27.535010 135501 storage/replica_raft.go:291 [n3,s3,r145/3:/Table/80/1/3{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 I190903 18:32:27.545691 135649 storage/store.go:2593 [n2,replicaGC,s2,r145/1:/Table/80/1/3{-/4}] removing replica r145/1 I190903 18:32:27.546099 135649 storage/replica_destroy.go:152 [n2,replicaGC,s2,r145/1:/Table/80/1/3{-/4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:27.655478 135960 storage/queue.go:1027 [n1,replicate,s1,r73/2:/Table/80/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:27.662388 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:27.800518 136171 storage/replica_command.go:1430 [n1,replicate,s1,r73/2:/Table/80/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r73:/Table/80/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=106] I190903 18:32:27.804248 136171 storage/replica_raft.go:291 [n1,s1,r73/2:/Table/80/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:27.813179 136332 storage/store.go:2593 [n3,replicaGC,s3,r73/1:/Table/80/1/{4-5/6}] removing replica r73/1 I190903 18:32:27.813411 136332 storage/replica_destroy.go:152 [n3,replicaGC,s3,r73/1:/Table/80/1/{4-5/6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:27.899072 136631 storage/queue.go:1027 [n2,replicate,s2,r71/3:/Table/80/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 18:32:27.933136 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:27.936348 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:27.939882 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:27.941393 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:27.944338 123380 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) === RUN TestRepartitioning/multi_col_range_partitioning_-_MAXVALUE/multi_col_range_partitioning I190903 18:32:27.960404 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=80 (multi col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 18:32:27.946923511 +0000 UTC I190903 18:32:27.968989 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 79, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning - MAXVALUE"]} I190903 18:32:27.971985 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 4}] I190903 18:32:28.023744 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=80 (multi col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 18:32:28.023569508 +0000 UTC I190903 18:32:28.043637 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 81, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:28.063959 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 82, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root} I190903 18:32:28.069522 138246 storage/replica_command.go:284 [n3,split,s3,r69/1:/{Table/80/2-Max}] initiating a split of this range at key /Table/82 [r74] (zone config) I190903 18:32:28.081147 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:28.085031 138354 storage/replica_command.go:284 [n3,split,s3,r74/1:/{Table/82-Max}] initiating a split of this range at key /Table/82/1 [r75] (zone config) I190903 18:32:28.100513 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:28.107241 138566 storage/replica_command.go:284 [n3,split,s3,r75/1:/{Table/82/1-Max}] initiating a split of this range at key /Table/82/1/3 [r76] (zone config) I190903 18:32:28.109642 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:28.120329 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:28.123105 139752 storage/replica_command.go:284 [n3,split,s3,r76/1:/{Table/82/1/3-Max}] initiating a split of this range at key /Table/82/1/4 [r77] (zone config) I190903 18:32:28.123929 139658 storage/replica_command.go:1430 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r75:/Table/82/1{-/3} [(n3,s3):1, next=2, gen=104] I190903 18:32:28.128668 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:28.128790 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:28.133803 139658 storage/replica_raft.go:291 [n3,s3,r75/1:/Table/82/1{-/3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:28.134932 139658 storage/store_snapshot.go:995 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] sending LEARNER snapshot 62ee8a37 at applied index 17 I190903 18:32:28.135378 139658 storage/store_snapshot.go:1038 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:28.136453 139817 storage/replica_raftstorage.go:793 [n2,s2,r75/2:{-}] applying LEARNER snapshot [id=62ee8a37 index=17] I190903 18:32:28.137968 139817 storage/replica_raftstorage.go:814 [n2,s2,r75/2:/Table/82/1{-/3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=62ee8a37 index=17] I190903 18:32:28.138771 139658 storage/replica_command.go:1430 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r75:/Table/82/1{-/3} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=105] I190903 18:32:28.140791 139873 storage/replica_command.go:284 [n3,split,s3,r77/1:/{Table/82/1/4-Max}] initiating a split of this range at key /Table/82/1/5/6 [r78] (zone config) I190903 18:32:28.144240 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:28.146807 139658 storage/replica_raft.go:291 [n3,s3,r75/1:/Table/82/1{-/3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:28.147246 139658 storage/replica_command.go:1430 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r75:/Table/82/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=106] I190903 18:32:28.148582 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.162084 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.164100 140348 storage/replica_command.go:284 [n3,split,s3,r78/1:/{Table/82/1/5…-Max}] initiating a split of this range at key /Table/82/1/5/7 [r79] (zone config) I190903 18:32:28.166693 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.169830 139658 storage/replica_raft.go:291 [n3,s3,r75/1:/Table/82/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:28.169900 139658 storage/replica_raft.go:304 [n3,s3,r75/1:/Table/82/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:28.170729 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.172089 139658 storage/replica_command.go:979 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] could not promote [] to voter, rolling back: change replicas of r75 failed: storage/replica_raft.go:305: [n3,s3,r75/1:/Table/82/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:28.172140 139658 storage/queue.go:1027 [n3,replicate,s3,r75/1:/Table/82/1{-/3}] change replicas of r75 failed: storage/replica_raft.go:305: [n3,s3,r75/1:/Table/82/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:28.172968 140368 storage/replica_command.go:1430 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r77:/Table/82/1/{4-5/6} [(n3,s3):1, next=2, gen=106] I190903 18:32:28.176053 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:28.177969 140189 storage/replica_command.go:284 [n3,split,s3,r79/1:/{Table/82/1/5…-Max}] initiating a split of this range at key /Table/82/2 [r80] (zone config) I190903 18:32:28.180751 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:28.182581 140368 storage/replica_raft.go:291 [n3,s3,r77/1:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:28.185248 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:28.188299 140368 storage/store_snapshot.go:995 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] sending LEARNER snapshot 7531141b at applied index 17 I190903 18:32:28.188447 140368 storage/store_snapshot.go:1038 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:28.189312 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.190853 140472 storage/replica_raftstorage.go:793 [n1,s1,r77/2:{-}] applying LEARNER snapshot [id=7531141b index=17] I190903 18:32:28.192230 140472 storage/replica_raftstorage.go:814 [n1,s1,r77/2:/Table/82/1/{4-5/6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=7531141b index=17] I190903 18:32:28.192631 140368 storage/replica_command.go:1430 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r77:/Table/82/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=107] I190903 18:32:28.193413 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:28.201095 140368 storage/replica_raft.go:291 [n3,s3,r77/1:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:28.201619 140368 storage/replica_command.go:1430 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r77:/Table/82/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=108] I190903 18:32:28.207616 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.219038 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.220017 140368 storage/replica_raft.go:291 [n3,s3,r77/1:/Table/82/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:28.220059 140368 storage/replica_raft.go:304 [n3,s3,r77/1:/Table/82/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:28.223195 140368 storage/replica_command.go:979 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r77 failed: storage/replica_raft.go:305: [n3,s3,r77/1:/Table/82/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:28.223268 140368 storage/queue.go:1027 [n3,replicate,s3,r77/1:/Table/82/1/{4-5/6}] change replicas of r77 failed: storage/replica_raft.go:305: [n3,s3,r77/1:/Table/82/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:28.223230 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:28.223751 140728 storage/replica_command.go:1430 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r78:/Table/82/1/5/{6-7} [(n3,s3):1, next=2, gen=107] I190903 18:32:28.226802 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.230266 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.240131 140728 storage/replica_raft.go:291 [n3,s3,r78/1:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:28.241306 140728 storage/store_snapshot.go:995 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] sending LEARNER snapshot 675f88aa at applied index 17 I190903 18:32:28.241453 140728 storage/store_snapshot.go:1038 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:28.242830 140733 storage/replica_raftstorage.go:793 [n2,s2,r78/2:{-}] applying LEARNER snapshot [id=675f88aa index=17] I190903 18:32:28.244110 140733 storage/replica_raftstorage.go:814 [n2,s2,r78/2:/Table/82/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=675f88aa index=17] I190903 18:32:28.244475 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:28.244610 140728 storage/replica_command.go:1430 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r78:/Table/82/1/5/{6-7} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=108] I190903 18:32:28.249669 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.252767 140728 storage/replica_raft.go:291 [n3,s3,r78/1:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:28.253331 140728 storage/replica_command.go:1430 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r78:/Table/82/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=109] I190903 18:32:28.254695 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.258185 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.261927 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.267939 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.268698 140728 storage/replica_raft.go:291 [n3,s3,r78/1:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:28.268793 140728 storage/replica_raft.go:304 [n3,s3,r78/1:/Table/82/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:28.272227 140728 storage/replica_command.go:979 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r78 failed: storage/replica_raft.go:305: [n3,s3,r78/1:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:28.272263 140728 storage/queue.go:1027 [n3,replicate,s3,r78/1:/Table/82/1/5/{6-7}] change replicas of r78 failed: storage/replica_raft.go:305: [n3,s3,r78/1:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:28.272779 140491 storage/replica_command.go:1430 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r80:/{Table/82/2-Max} [(n3,s3):1, next=2, gen=108] I190903 18:32:28.273335 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:28.279635 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.280203 140491 storage/replica_raft.go:291 [n3,s3,r80/1:/{Table/82/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:28.284667 140491 storage/store_snapshot.go:995 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] sending LEARNER snapshot 4ea53c62 at applied index 14 I190903 18:32:28.284964 140491 storage/store_snapshot.go:1038 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:28.286338 140351 storage/replica_raftstorage.go:793 [n1,s1,r80/2:{-}] applying LEARNER snapshot [id=4ea53c62 index=14] I190903 18:32:28.294865 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:28.295528 140351 storage/replica_raftstorage.go:814 [n1,s1,r80/2:/{Table/82/2-Max}] applied LEARNER snapshot [total=9ms ingestion=4@1ms id=4ea53c62 index=14] I190903 18:32:28.296049 140491 storage/replica_command.go:1430 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r80:/{Table/82/2-Max} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=109] I190903 18:32:28.303884 140491 storage/replica_raft.go:291 [n3,s3,r80/1:/{Table/82/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:28.304691 140491 storage/replica_command.go:1430 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r80:/{Table/82/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=110] I190903 18:32:28.307224 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.315715 141035 storage/replica_command.go:1430 [n2,replicate,s2,r71/3:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r71:/Table/80/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=109] I190903 18:32:28.318340 140491 storage/replica_raft.go:291 [n3,s3,r80/1:/{Table/82/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:28.318399 140491 storage/replica_raft.go:304 [n3,s3,r80/1:/{Table/82/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:28.319420 140491 storage/replica_command.go:979 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] could not promote [] to voter, rolling back: change replicas of r80 failed: storage/replica_raft.go:305: [n3,s3,r80/1:/{Table/82/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:28.319459 140491 storage/queue.go:1027 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] change replicas of r80 failed: storage/replica_raft.go:305: [n3,s3,r80/1:/{Table/82/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:28.320247 141178 storage/replica_command.go:1430 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r79:/Table/82/{1/5/7-2} [(n3,s3):1, next=2, gen=108] I190903 18:32:28.321874 141035 storage/replica_raft.go:291 [n2,s2,r71/3:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:28.326995 141178 storage/replica_raft.go:291 [n3,s3,r79/1:/Table/82/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:28.328089 141178 storage/store_snapshot.go:995 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] sending LEARNER snapshot 89329908 at applied index 18 I190903 18:32:28.328143 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.328253 141178 storage/store_snapshot.go:1038 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:28.329599 141226 storage/replica_raftstorage.go:793 [n1,s1,r79/2:{-}] applying LEARNER snapshot [id=89329908 index=18] I190903 18:32:28.329982 140992 storage/store.go:2593 [n1,replicaGC,s1,r71/2:/Table/80/1/5/{6-7}] removing replica r71/2 I190903 18:32:28.330575 140992 storage/replica_destroy.go:152 [n1,replicaGC,s1,r71/2:/Table/80/1/5/{6-7}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:28.331631 141226 storage/replica_raftstorage.go:814 [n1,s1,r79/2:/Table/82/{1/5/7-2}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=89329908 index=18] I190903 18:32:28.332287 141178 storage/replica_command.go:1430 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r79:/Table/82/{1/5/7-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=109] I190903 18:32:28.343875 141178 storage/replica_raft.go:291 [n3,s3,r79/1:/Table/82/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:28.344529 141178 storage/replica_command.go:1430 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r79:/Table/82/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=110] I190903 18:32:28.353040 141178 storage/replica_raft.go:291 [n3,s3,r79/1:/Table/82/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:28.353084 141178 storage/replica_raft.go:304 [n3,s3,r79/1:/Table/82/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:28.354175 141178 storage/replica_command.go:979 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r79 failed: storage/replica_raft.go:305: [n3,s3,r79/1:/Table/82/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:28.354214 141178 storage/queue.go:1027 [n3,replicate,s3,r79/1:/Table/82/{1/5/7-2}] change replicas of r79 failed: storage/replica_raft.go:305: [n3,s3,r79/1:/Table/82/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:28.354633 141357 storage/replica_command.go:1430 [n3,replicate,s3,r86/2:/Table/74{-/1}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r86:/Table/74{-/1} [(n3,s3):2, next=3, gen=83] I190903 18:32:28.365582 141357 storage/replica_raft.go:291 [n3,s3,r86/2:/Table/74{-/1}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 18:32:28.366544 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.368129 141357 storage/store_snapshot.go:995 [n3,replicate,s3,r86/2:/Table/74{-/1}] sending LEARNER snapshot b73f0b24 at applied index 52 I190903 18:32:28.368282 141357 storage/store_snapshot.go:1038 [n3,replicate,s3,r86/2:/Table/74{-/1}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:28.369553 141263 storage/replica_raftstorage.go:793 [n1,s1,r86/3:{-}] applying LEARNER snapshot [id=b73f0b24 index=52] I190903 18:32:28.373621 141263 storage/replica_raftstorage.go:814 [n1,s1,r86/3:/Table/74{-/1}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=b73f0b24 index=52] I190903 18:32:28.374531 141357 storage/replica_command.go:1430 [n3,replicate,s3,r86/2:/Table/74{-/1}] change replicas (add [(n1,s1):3] remove []): existing descriptor r86:/Table/74{-/1} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=84] I190903 18:32:28.382584 141357 storage/replica_raft.go:291 [n3,s3,r86/2:/Table/74{-/1}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 18:32:28.383643 141357 storage/replica_command.go:1430 [n3,replicate,s3,r86/2:/Table/74{-/1}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r86:/Table/74{-/1} [(n3,s3):2, (n1,s1):3, next=4, gen=85] I190903 18:32:28.405949 141357 storage/replica_raft.go:291 [n3,s3,r86/2:/Table/74{-/1}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 18:32:28.405982 141357 storage/replica_raft.go:304 [n3,s3,r86/2:/Table/74{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:28.411528 141357 storage/replica_command.go:979 [n3,replicate,s3,r86/2:/Table/74{-/1}] could not promote [] to voter, rolling back: change replicas of r86 failed: storage/replica_raft.go:305: [n3,s3,r86/2:/Table/74{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:28.411560 141357 storage/queue.go:1027 [n3,replicate,s3,r86/2:/Table/74{-/1}] change replicas of r86 failed: storage/replica_raft.go:305: [n3,s3,r86/2:/Table/74{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:28.412016 141518 storage/replica_command.go:1430 [n3,replicate,s3,r28/1:/Table/62{-/1}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r28:/Table/62{-/1} [(n3,s3):1, next=3, gen=25] I190903 18:32:28.420645 141518 storage/replica_raft.go:291 [n3,s3,r28/1:/Table/62{-/1}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):1 (n1,s1):3LEARNER] next=4 I190903 18:32:28.421743 141518 storage/store_snapshot.go:995 [n3,replicate,s3,r28/1:/Table/62{-/1}] sending LEARNER snapshot 7281e7ca at applied index 30 I190903 18:32:28.421894 141518 storage/store_snapshot.go:1038 [n3,replicate,s3,r28/1:/Table/62{-/1}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:28.423014 141457 storage/replica_raftstorage.go:793 [n1,s1,r28/3:{-}] applying LEARNER snapshot [id=7281e7ca index=30] I190903 18:32:28.425273 141457 storage/replica_raftstorage.go:814 [n1,s1,r28/3:/Table/62{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=7281e7ca index=30] I190903 18:32:28.427650 141518 storage/replica_command.go:1430 [n3,replicate,s3,r28/1:/Table/62{-/1}] change replicas (add [(n1,s1):3] remove []): existing descriptor r28:/Table/62{-/1} [(n3,s3):1, (n1,s1):3LEARNER, next=4, gen=26] I190903 18:32:28.435900 141518 storage/replica_raft.go:291 [n3,s3,r28/1:/Table/62{-/1}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):1 (n1,s1):3] next=4 I190903 18:32:28.437225 141518 storage/replica_command.go:1430 [n3,replicate,s3,r28/1:/Table/62{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r28:/Table/62{-/1} [(n3,s3):1, (n1,s1):3, next=4, gen=27] I190903 18:32:28.439323 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:28.447333 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.451170 141518 storage/replica_raft.go:291 [n3,s3,r28/1:/Table/62{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 E190903 18:32:28.451232 141518 storage/replica_raft.go:304 [n3,s3,r28/1:/Table/62{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:28.456373 141518 storage/replica_command.go:979 [n3,replicate,s3,r28/1:/Table/62{-/1}] could not promote [] to voter, rolling back: change replicas of r28 failed: storage/replica_raft.go:305: [n3,s3,r28/1:/Table/62{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:28.456471 141518 storage/queue.go:1027 [n3,replicate,s3,r28/1:/Table/62{-/1}] change replicas of r28 failed: storage/replica_raft.go:305: [n3,s3,r28/1:/Table/62{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:28.516912 141936 storage/queue.go:1027 [n2,replicate,s2,r75/2:/Table/82/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:24] E190903 18:32:28.557549 142053 storage/queue.go:1027 [n1,replicate,s1,r28/3:/Table/62{-/1}] no removable replicas from range that needs a removal: [1:0, 3*:37] I190903 18:32:28.587038 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) E190903 18:32:28.711565 142431 storage/queue.go:1027 [n1,replicate,s1,r77/2:/Table/82/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:28.822021 142799 storage/replica_command.go:1430 [n2,replicate,s2,r75/2:/Table/82/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r75:/Table/82/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=106] E190903 18:32:28.825504 142765 storage/queue.go:1027 [n1,replicate,s1,r86/3:/Table/74{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:59] I190903 18:32:28.832944 142799 storage/replica_raft.go:291 [n2,s2,r75/2:/Table/82/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:28.841862 142770 storage/store.go:2593 [n3,replicaGC,s3,r75/1:/Table/82/1{-/3}] removing replica r75/1 I190903 18:32:28.842244 142770 storage/replica_destroy.go:152 [n3,replicaGC,s3,r75/1:/Table/82/1{-/3}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:28.859152 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:28.862673 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:28.865521 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:28.922540 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:28.931328 143047 storage/replica_command.go:1430 [n1,replicate,s1,r28/3:/Table/62{-/1}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r28:/Table/62{-/1} [(n3,s3):1, (n1,s1):3, next=4, gen=27] I190903 18:32:28.934696 143047 storage/replica_raft.go:291 [n1,s1,r28/3:/Table/62{-/1}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 I190903 18:32:28.937554 141818 storage/replica_command.go:1430 [n1,replicate,s1,r79/2:/Table/82/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r79:/Table/82/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=110] I190903 18:32:28.944680 141818 storage/replica_raft.go:291 [n1,s1,r79/2:/Table/82/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:28.949874 142904 storage/store.go:2593 [n3,replicaGC,s3,r28/1:/Table/62{-/1}] removing replica r28/1 I190903 18:32:28.950273 142904 storage/replica_destroy.go:152 [n3,replicaGC,s3,r28/1:/Table/62{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:28.957992 143185 storage/store.go:2593 [n3,replicaGC,s3,r79/1:/Table/82/{1/5/7-2}] removing replica r79/1 I190903 18:32:28.958480 143185 storage/replica_destroy.go:152 [n3,replicaGC,s3,r79/1:/Table/82/{1/5/7-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:28.985489 143050 storage/consistency_queue.go:133 [n3,consistencyChecker,s3,r79/1:/Table/82/{1/5/7-2}] could not get replica descriptor: r79 was not found on s3 E190903 18:32:28.985553 143050 storage/queue.go:1027 [n3,consistencyChecker,s3,r79/1:/Table/82/{1/5/7-2}] could not get replica descriptor: r79 was not found on s3 E190903 18:32:29.008040 143428 storage/queue.go:1027 [n1,replicate,s1,r80/2:/{Table/82/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:23] E190903 18:32:29.013504 143248 storage/queue.go:1027 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:29.027967 143486 storage/replica_command.go:609 [n3,merge,s3,r145/3:/Table/80/1/3{-/4}] initiating a merge of r70:/Table/80/1/{3/4-4} [(n3,s3):1, next=3, gen=107] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:29.038700 2304 storage/store.go:2593 [n3,s3,r145/3:/Table/80/1/3{-/4}] removing replica r70/1 I190903 18:32:29.039916 143533 storage/replica_command.go:1430 [n1,replicate,s1,r77/2:/Table/82/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r77:/Table/82/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=108] I190903 18:32:29.045864 143533 storage/replica_raft.go:291 [n1,s1,r77/2:/Table/82/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:29.055907 143512 storage/store.go:2593 [n3,replicaGC,s3,r77/1:/Table/82/1/{4-5/6}] removing replica r77/1 I190903 18:32:29.056282 143512 storage/replica_destroy.go:152 [n3,replicaGC,s3,r77/1:/Table/82/1/{4-5/6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:29.070377 143619 storage/replica_command.go:1430 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r78:/Table/82/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=109] I190903 18:32:29.079070 143619 storage/replica_raft.go:291 [n2,s2,r78/2:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:29.089171 143694 storage/store.go:2593 [n3,replicaGC,s3,r78/1:/Table/82/1/5/{6-7}] removing replica r78/1 I190903 18:32:29.089553 143694 storage/replica_destroy.go:152 [n3,replicaGC,s3,r78/1:/Table/82/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:29.168539 143960 storage/queue.go:1027 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] no removable replicas from range that needs a removal: [1*:25, 2:0] I190903 18:32:29.185801 143857 storage/replica_command.go:1430 [n3,replicate,s3,r80/1:/{Table/82/2-Max}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r80:/{Table/82/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=110] I190903 18:32:29.196937 143857 storage/replica_raft.go:291 [n3,s3,r80/1:/{Table/82/2-Max}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 18:32:29.205122 144000 storage/store.go:2593 [n1,replicaGC,s1,r80/2:/{Table/82/2-Max}] removing replica r80/2 I190903 18:32:29.205752 144000 storage/replica_destroy.go:152 [n1,replicaGC,s1,r80/2:/{Table/82/2-Max}] removed 8 (0+8) keys in 1ms [clear=0ms commit=0ms] I190903 18:32:29.464110 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:29.467656 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:29.470859 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:29.472532 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:29.476800 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:29.481221 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=82 (multi col range partitioning) version=2 mtime=2019-09-03 18:32:29.479445804 +0000 UTC I190903 18:32:29.489633 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 1}] I190903 18:32:29.492587 144952 sql/lease.go:923 new lease: 82("multi col range partitioning") ver=2:1567535862.058338059,0, refcount=0 I190903 18:32:29.549022 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=82 (multi col range partitioning) version=3 mtime=2019-09-03 18:32:29.548829565 +0000 UTC I190903 18:32:29.572913 145359 sql/lease.go:923 new lease: 82("multi col range partitioning") ver=3:1567535862.059338059,0, refcount=0 I190903 18:32:29.575069 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 2}] I190903 18:32:29.602150 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=82 (multi col range partitioning) version=4 mtime=2019-09-03 18:32:29.598905328 +0000 UTC E190903 18:32:29.609779 145824 storage/queue.go:1027 [n3,replicate,s3,r86/2:/Table/74{-/1}] no removable replicas from range that needs a removal: [2*:64, 3:0] I190903 18:32:29.614350 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 82, info: {TableName:data.public."multi col range partitioning" Statement:ALTER TABLE "multi col range partitioning" PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p56 VALUES FROM (3, 4) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:29.616500 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 3}] I190903 18:32:29.619179 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:29.622657 145795 sql/lease.go:923 new lease: 82("multi col range partitioning") ver=4:1567535862.060338059,0, refcount=0 E190903 18:32:29.623855 146051 storage/queue.go:1027 [n2,merge,s2,r75/2:/Table/82/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 18:32:29.662975 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:29.669041 146594 storage/replica_command.go:284 [n3,split,s3,r76/1:/Table/82/1/{3-4}] initiating a split of this range at key /Table/82/1/3/4 [r161] (zone config) I190903 18:32:29.680081 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:29.686121 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:29.687382 146643 storage/replica_command.go:1430 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r161:/Table/82/1/{3/4-4} [(n3,s3):1, next=2, gen=106] E190903 18:32:29.688096 146928 storage/queue.go:1027 [n3,merge,s3,r161/1:/Table/82/1/{3/4-4}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 18:32:29.688967 146982 storage/replica_command.go:1430 [n3,s3,r76/1:/Table/82/1/3{-/4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r76:/Table/82/1/3{-/4} [(n3,s3):1, next=2, gen=106] I190903 18:32:29.696999 146643 storage/replica_raft.go:291 [n3,s3,r161/1:/Table/82/1/{3/4-4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:29.698772 146643 storage/store_snapshot.go:995 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] sending LEARNER snapshot 69670601 at applied index 14 I190903 18:32:29.699065 146643 storage/store_snapshot.go:1038 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:29.700305 147076 storage/replica_raftstorage.go:793 [n1,s1,r161/2:{-}] applying LEARNER snapshot [id=69670601 index=14] I190903 18:32:29.710058 147076 storage/replica_raftstorage.go:814 [n1,s1,r161/2:/Table/82/1/{3/4-4}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=69670601 index=14] I190903 18:32:29.711130 146643 storage/replica_command.go:1430 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=107] I190903 18:32:29.712338 146982 storage/replica_raft.go:291 [n3,s3,r76/1:/Table/82/1/3{-/4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:29.713769 146982 storage/store_snapshot.go:995 [n3,s3,r76/1:/Table/82/1/3{-/4}] sending LEARNER snapshot f828a10e at applied index 27 I190903 18:32:29.713968 146982 storage/store_snapshot.go:1038 [n3,s3,r76/1:/Table/82/1/3{-/4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:29.715087 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:29.715198 147080 storage/replica_raftstorage.go:793 [n2,s2,r76/2:{-}] applying LEARNER snapshot [id=f828a10e index=27] I190903 18:32:29.717119 147080 storage/replica_raftstorage.go:814 [n2,s2,r76/2:/Table/82/1/3{-/4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=f828a10e index=27] I190903 18:32:29.723664 146982 storage/replica_command.go:1430 [n3,s3,r76/1:/Table/82/1/3{-/4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r76:/Table/82/1/3{-/4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=107] I190903 18:32:29.725094 146643 storage/replica_raft.go:291 [n3,s3,r161/1:/Table/82/1/{3/4-4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:29.726092 146643 storage/replica_command.go:1430 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=108] I190903 18:32:29.726399 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:29.726607 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:29.734914 146989 storage/replica_command.go:979 [n3,s3,r161/1:/Table/82/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r161 failed: descriptor changed: [expected] r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=107] != [actual] r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=108] E190903 18:32:29.734997 146989 storage/queue.go:1027 [n3,merge,s3,r161/1:/Table/82/1/{3/4-4}] removing learners from r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=107]: change replicas of r161 failed: descriptor changed: [expected] r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=107] != [actual] r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=108] I190903 18:32:29.737456 146094 storage/replica_command.go:1430 [n1,s1,r77/2:/Table/82/1/{4-5/6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r77:/Table/82/1/{4-5/6} [(n1,s1):2, next=3, gen=109] I190903 18:32:29.737769 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:29.739265 146982 storage/replica_raft.go:291 [n3,s3,r76/1:/Table/82/1/3{-/4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:29.740737 146643 storage/replica_raft.go:291 [n3,s3,r161/1:/Table/82/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:29.740785 146643 storage/replica_raft.go:304 [n3,s3,r161/1:/Table/82/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:29.741302 146094 storage/replica_raft.go:291 [n1,s1,r77/2:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:29.743465 146094 storage/store_snapshot.go:995 [n1,s1,r77/2:/Table/82/1/{4-5/6}] sending LEARNER snapshot 0c86fd44 at applied index 31 I190903 18:32:29.743637 146094 storage/store_snapshot.go:1038 [n1,s1,r77/2:/Table/82/1/{4-5/6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:29.743939 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:29.744825 147914 storage/replica_raftstorage.go:793 [n3,s3,r77/3:{-}] applying LEARNER snapshot [id=0c86fd44 index=31] I190903 18:32:29.748936 146643 storage/replica_command.go:979 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r161 failed: storage/replica_raft.go:305: [n3,s3,r161/1:/Table/82/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:29.749027 146643 storage/queue.go:1027 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] change replicas of r161 failed: storage/replica_raft.go:305: [n3,s3,r161/1:/Table/82/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:29.753173 147906 storage/replica_command.go:1430 [n3,replicate,s3,r161/1:/Table/82/1/{3/4-4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r161:/Table/82/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=108] I190903 18:32:29.760960 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 18:32:29.760981 148053 storage/queue.go:1027 [n2,replicate,s2,r76/2:/Table/82/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:31] I190903 18:32:29.765141 147914 storage/replica_raftstorage.go:814 [n3,s3,r77/3:/Table/82/1/{4-5/6}] applied LEARNER snapshot [total=20ms ingestion=4@20ms id=0c86fd44 index=31] I190903 18:32:29.765647 146094 storage/replica_command.go:1430 [n1,s1,r77/2:/Table/82/1/{4-5/6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r77:/Table/82/1/{4-5/6} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=110] I190903 18:32:29.769373 146094 storage/replica_raft.go:291 [n1,s1,r77/2:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 18:32:29.771926 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:29.787758 147906 storage/replica_raft.go:291 [n3,s3,r161/1:/Table/82/1/{3/4-4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 18:32:29.790533 148182 storage/replica_command.go:1430 [n3,replicate,s3,r77/3:/Table/82/1/{4-5/6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r77:/Table/82/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=111] I190903 18:32:29.794361 148196 storage/replica_command.go:1430 [n2,s2,r76/2:/Table/82/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r76:/Table/82/1/3{-/4} [(n3,s3):1, (n2,s2):2, next=3, gen=108] I190903 18:32:29.796543 147856 storage/store.go:2593 [n1,replicaGC,s1,r161/2:/Table/82/1/{3/4-4}] removing replica r161/2 I190903 18:32:29.796848 147856 storage/replica_destroy.go:152 [n1,replicaGC,s1,r161/2:/Table/82/1/{3/4-4}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:29.799110 148182 storage/replica_raft.go:291 [n3,s3,r77/3:/Table/82/1/{4-5/6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:29.807745 147756 storage/store.go:2593 [n1,replicaGC,s1,r77/2:/Table/82/1/{4-5/6}] removing replica r77/2 I190903 18:32:29.808365 147756 storage/replica_destroy.go:152 [n1,replicaGC,s1,r77/2:/Table/82/1/{4-5/6}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:29.809777 148196 storage/replica_raft.go:291 [n2,s2,r76/2:/Table/82/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:29.812780 146841 storage/replica_command.go:609 [n2,merge,s2,r75/2:/Table/82/1{-/3}] initiating a merge of r76:/Table/82/1/3{-/4} [(n2,s2):2, next=3, gen=109] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:29.821199 148301 storage/store.go:2593 [n3,replicaGC,s3,r76/1:/Table/82/1/3{-/4}] removing replica r76/1 I190903 18:32:29.821579 148301 storage/replica_destroy.go:152 [n3,replicaGC,s3,r76/1:/Table/82/1/3{-/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:29.831950 146093 storage/replica_command.go:609 [n3,merge,s3,r161/1:/Table/82/1/{3/4-4}] initiating a merge of r77:/Table/82/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=111] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:29.834402 1538 storage/store.go:2593 [n2,s2,r75/2:/Table/82/1{-/3}] removing replica r76/2 I190903 18:32:29.947716 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:29.950833 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:29.956103 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:29.959700 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:29.961976 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:29.965070 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:29.968841 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:29.975426 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:29.979332 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.001658 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.012746 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.016380 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.024003 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.029939 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.037269 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.040311 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.045140 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.048533 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.052921 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.056453 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.059794 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:30.061698 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.067574 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.071225 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.075059 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.078383 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.082202 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.085667 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.090743 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:30.092589 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.095397 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.099301 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.102366 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.105608 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.109318 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.112338 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 18:32:30.112639 149358 storage/queue.go:1027 [n1,replicate,s1,r86/3:/Table/74{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:67] I190903 18:32:30.115491 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.118770 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.122132 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.135595 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.139630 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.143628 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.147147 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.155610 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.159218 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.162748 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.165935 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.169717 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.175126 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.181592 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.188838 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.218961 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.229169 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.232914 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.240129 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.243519 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.249703 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.262291 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.282685 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.290437 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.305375 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.312615 149922 storage/replica_command.go:1430 [n1,replicate,s1,r86/3:/Table/74{-/1}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r86:/Table/74{-/1} [(n3,s3):2, (n1,s1):3, next=4, gen=85] I190903 18:32:30.317433 150182 storage/replica_command.go:1430 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r78:/Table/82/1/5/{6-7} [(n2,s2):2, next=3, gen=110] I190903 18:32:30.317914 149922 storage/replica_raft.go:291 [n1,s1,r86/3:/Table/74{-/1}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 18:32:30.327429 150141 storage/store.go:2593 [n3,replicaGC,s3,r86/2:/Table/74{-/1}] removing replica r86/2 I190903 18:32:30.327661 150182 storage/replica_raft.go:291 [n2,s2,r78/2:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:30.327898 150141 storage/replica_destroy.go:152 [n3,replicaGC,s3,r86/2:/Table/74{-/1}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:30.329729 150182 storage/store_snapshot.go:995 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] sending LEARNER snapshot e365c354 at applied index 35 I190903 18:32:30.329913 150182 storage/store_snapshot.go:1038 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:30.329973 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.331543 150239 storage/replica_raftstorage.go:793 [n1,s1,r78/3:{-}] applying LEARNER snapshot [id=e365c354 index=35] I190903 18:32:30.333451 150239 storage/replica_raftstorage.go:814 [n1,s1,r78/3:/Table/82/1/5/{6-7}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=e365c354 index=35] I190903 18:32:30.335629 150182 storage/replica_command.go:1430 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r78:/Table/82/1/5/{6-7} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=111] I190903 18:32:30.340249 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.346679 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.347659 150182 storage/replica_raft.go:291 [n2,s2,r78/2:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:30.348281 150182 storage/replica_command.go:1430 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r78:/Table/82/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=112] I190903 18:32:30.371659 150182 storage/replica_raft.go:291 [n2,s2,r78/2:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:30.371710 150182 storage/replica_raft.go:304 [n2,s2,r78/2:/Table/82/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:30.375442 150182 storage/replica_command.go:979 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r78 failed: storage/replica_raft.go:305: [n2,s2,r78/2:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:30.375516 150182 storage/queue.go:1027 [n2,replicate,s2,r78/2:/Table/82/1/5/{6-7}] change replicas of r78 failed: storage/replica_raft.go:305: [n2,s2,r78/2:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:30.395579 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:30.404924 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.418859 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.423101 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.493700 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.500544 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.504767 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:30.642678 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.646316 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 18:32:30.891099 151769 storage/queue.go:1027 [n1,replicate,s1,r78/3:/Table/82/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 18:32:30.920096 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 18:32:30.922987 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 18:32:30.926336 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 18:32:30.940387 136569 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) === RUN TestRepartitioning/multi_col_range_partitioning_-_MAXVALUE/multi_col_range_partitioning_-_MAXVALUE_MAXVALUE I190903 18:32:31.156585 152554 storage/replica_command.go:1430 [n1,replicate,s1,r78/3:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r78:/Table/82/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=112] I190903 18:32:31.159407 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=82 (multi col range partitioning) version=5 mtime=2019-09-03 18:32:30.995731186 +0000 UTC I190903 18:32:31.164052 152554 storage/replica_raft.go:291 [n1,s1,r78/3:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:31.182072 152677 storage/store.go:2593 [n2,replicaGC,s2,r78/2:/Table/82/1/5/{6-7}] removing replica r78/2 I190903 18:32:31.182811 152677 storage/replica_destroy.go:152 [n2,replicaGC,s2,r78/2:/Table/82/1/5/{6-7}] removed 8 (0+8) keys in 1ms [clear=0ms commit=0ms] I190903 18:32:31.216669 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:31.219760 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 81, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning"]} I190903 18:32:31.233927 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=82 (multi col range partitioning) version=6 mtime=2019-09-03 18:32:31.233369208 +0000 UTC I190903 18:32:31.367927 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 83, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:31.388089 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 84, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root} I190903 18:32:31.391643 154487 storage/replica_command.go:284 [n3,split,s3,r80/1:/{Table/82/2-Max}] initiating a split of this range at key /Table/84 [r162] (zone config) I190903 18:32:31.442054 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:31.446761 155038 storage/replica_command.go:284 [n3,split,s3,r162/1:/{Table/84-Max}] initiating a split of this range at key /Table/84/1 [r163] (zone config) I190903 18:32:31.464593 155208 storage/replica_command.go:284 [n3,split,s3,r163/1:/{Table/84/1-Max}] initiating a split of this range at key /Table/84/2 [r164] (zone config) I190903 18:32:31.477873 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:31.480203 2232 storage/queue.go:518 [n3,s3,r163/1:/Table/84/{1-2}] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:31.482380 155566 storage/replica_command.go:284 [n3,split,s3,r163/1:/Table/84/{1-2}] initiating a split of this range at key /Table/84/1/3 [r165] (zone config) I190903 18:32:31.517926 155717 storage/replica_command.go:1430 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r163:/Table/84/1{-/3} [(n3,s3):1, next=2, gen=115] I190903 18:32:31.520959 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:31.522843 155228 storage/replica_command.go:284 [n3,split,s3,r165/1:/Table/84/{1/3-2}] initiating a split of this range at key /Table/84/1/4 [r166] (zone config) I190903 18:32:31.611369 155717 storage/replica_raft.go:291 [n3,s3,r163/1:/Table/84/1{-/3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:31.630696 155717 storage/store_snapshot.go:995 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] sending LEARNER snapshot 7e3c416d at applied index 21 I190903 18:32:31.630885 155717 storage/store_snapshot.go:1038 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:31.632867 156294 storage/replica_raftstorage.go:793 [n2,s2,r163/2:{-}] applying LEARNER snapshot [id=7e3c416d index=21] I190903 18:32:31.637572 156294 storage/replica_raftstorage.go:814 [n2,s2,r163/2:/Table/84/1{-/3}] applied LEARNER snapshot [total=5ms ingestion=4@1ms id=7e3c416d index=21] I190903 18:32:31.638944 155717 storage/replica_command.go:1430 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r163:/Table/84/1{-/3} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=116] I190903 18:32:31.648232 155717 storage/replica_raft.go:291 [n3,s3,r163/1:/Table/84/1{-/3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:31.648779 155717 storage/replica_command.go:1430 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r163:/Table/84/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=117] I190903 18:32:31.653116 124 storage/queue.go:518 [n1,s1,r8/1:/Table/1{2-3}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:31.662977 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:31.663873 156742 storage/replica_command.go:284 [n3,split,s3,r166/1:/Table/84/{1/4-2}] initiating a split of this range at key /Table/84/1/5/6 [r167] (zone config) I190903 18:32:31.668723 155717 storage/replica_raft.go:291 [n3,s3,r163/1:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:31.668785 155717 storage/replica_raft.go:304 [n3,s3,r163/1:/Table/84/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:31.670898 155717 storage/replica_command.go:979 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] could not promote [] to voter, rolling back: change replicas of r163 failed: storage/replica_raft.go:305: [n3,s3,r163/1:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:31.670968 155717 storage/queue.go:1027 [n3,replicate,s3,r163/1:/Table/84/1{-/3}] change replicas of r163 failed: storage/replica_raft.go:305: [n3,s3,r163/1:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:31.673297 156791 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r166:/Table/84/1/{4-5/6} [(n3,s3):1, next=2, gen=117] I190903 18:32:31.694255 156791 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:31.697293 156791 storage/store_snapshot.go:995 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] sending LEARNER snapshot fcdd9669 at applied index 18 I190903 18:32:31.697438 156791 storage/store_snapshot.go:1038 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:31.698762 156863 storage/replica_raftstorage.go:793 [n1,s1,r166/2:{-}] applying LEARNER snapshot [id=fcdd9669 index=18] I190903 18:32:31.700856 156863 storage/replica_raftstorage.go:814 [n1,s1,r166/2:/Table/84/1/{4-5/6}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=fcdd9669 index=18] I190903 18:32:31.701311 156791 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r166:/Table/84/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=118] I190903 18:32:31.727783 156791 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:31.735312 156791 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r166:/Table/84/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=119] I190903 18:32:31.742796 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:31.743071 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.746740 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:31.750588 157266 storage/replica_command.go:284 [n3,split,s3,r167/1:/Table/84/{1/5/6-2}] initiating a split of this range at key /Table/84/1/5/7 [r168] (zone config) I190903 18:32:31.761312 156791 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:31.761397 156791 storage/replica_raft.go:304 [n3,s3,r166/1:/Table/84/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:31.769021 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:31.772602 156791 storage/replica_command.go:979 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r166 failed: storage/replica_raft.go:305: [n3,s3,r166/1:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:31.775634 156791 storage/queue.go:1027 [n3,replicate,s3,r166/1:/Table/84/1/{4-5/6}] change replicas of r166 failed: storage/replica_raft.go:305: [n3,s3,r166/1:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:31.776153 157485 storage/replica_command.go:1430 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r167:/Table/84/1/5/{6-7} [(n3,s3):1, next=2, gen=118] I190903 18:32:31.781098 157485 storage/replica_raft.go:291 [n3,s3,r167/1:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:31.782269 157485 storage/store_snapshot.go:995 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] sending LEARNER snapshot ef6dd21c at applied index 18 I190903 18:32:31.782422 157485 storage/store_snapshot.go:1038 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:31.784787 157574 storage/replica_raftstorage.go:793 [n2,s2,r167/2:{-}] applying LEARNER snapshot [id=ef6dd21c index=18] I190903 18:32:31.786154 157574 storage/replica_raftstorage.go:814 [n2,s2,r167/2:/Table/84/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ef6dd21c index=18] I190903 18:32:31.786670 157485 storage/replica_command.go:1430 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r167:/Table/84/1/5/{6-7} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=119] I190903 18:32:31.787182 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:31.795381 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.801232 157485 storage/replica_raft.go:291 [n3,s3,r167/1:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:31.802299 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:31.802334 157485 storage/replica_command.go:1430 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r167:/Table/84/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=120] I190903 18:32:31.807580 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:31.818246 157485 storage/replica_raft.go:291 [n3,s3,r167/1:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:31.818332 157485 storage/replica_raft.go:304 [n3,s3,r167/1:/Table/84/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:31.819985 157485 storage/replica_command.go:979 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r167 failed: storage/replica_raft.go:305: [n3,s3,r167/1:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:31.820037 157485 storage/queue.go:1027 [n3,replicate,s3,r167/1:/Table/84/1/5/{6-7}] change replicas of r167 failed: storage/replica_raft.go:305: [n3,s3,r167/1:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:31.821334 157026 storage/replica_command.go:1430 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r168:/Table/84/{1/5/7-2} [(n3,s3):1, next=2, gen=118] I190903 18:32:31.821591 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:31.836840 157026 storage/replica_raft.go:291 [n3,s3,r168/1:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:31.838554 157026 storage/store_snapshot.go:995 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] sending LEARNER snapshot a9847617 at applied index 14 I190903 18:32:31.838698 157026 storage/store_snapshot.go:1038 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:31.840146 157641 storage/replica_raftstorage.go:793 [n1,s1,r168/2:{-}] applying LEARNER snapshot [id=a9847617 index=14] I190903 18:32:31.847149 157641 storage/replica_raftstorage.go:814 [n1,s1,r168/2:/Table/84/{1/5/7-2}] applied LEARNER snapshot [total=7ms ingestion=4@1ms id=a9847617 index=14] I190903 18:32:31.847690 157026 storage/replica_command.go:1430 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r168:/Table/84/{1/5/7-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119] I190903 18:32:31.847752 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:31.857218 157026 storage/replica_raft.go:291 [n3,s3,r168/1:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:31.859013 157026 storage/replica_command.go:1430 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r168:/Table/84/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 18:32:31.862536 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.873637 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.878827 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.879805 157026 storage/replica_raft.go:291 [n3,s3,r168/1:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:31.879871 157026 storage/replica_raft.go:304 [n3,s3,r168/1:/Table/84/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:31.880728 157026 storage/replica_command.go:979 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r168 failed: storage/replica_raft.go:305: [n3,s3,r168/1:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:31.880763 157026 storage/queue.go:1027 [n3,replicate,s3,r168/1:/Table/84/{1/5/7-2}] change replicas of r168 failed: storage/replica_raft.go:305: [n3,s3,r168/1:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:31.882217 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.885576 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.893299 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:31.896911 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:31.903532 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:31.906958 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.910332 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.913697 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:31.917335 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:31.922934 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.934147 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.938847 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:31.943576 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) E190903 18:32:31.946484 157886 storage/queue.go:1027 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:31.947648 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.952742 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.961850 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.976007 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:31.991745 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:32.012735 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:32.052859 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:32.123562 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 18:32:32.183520 158872 storage/queue.go:1027 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 18:32:32.204984 158850 storage/queue.go:1027 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:21] I190903 18:32:32.261456 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 18:32:32.269005 159160 storage/queue.go:1027 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:28] I190903 18:32:32.459698 159222 storage/replica_command.go:1430 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r167:/Table/84/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=120] I190903 18:32:32.525969 159222 storage/replica_raft.go:291 [n2,s2,r167/2:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:32.533530 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:32.582335 157979 storage/store.go:2593 [n3,replicaGC,s3,r167/1:/Table/84/1/5/{6-7}] removing replica r167/1 I190903 18:32:32.582790 157979 storage/replica_destroy.go:152 [n3,replicaGC,s3,r167/1:/Table/84/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:32.727917 160533 storage/replica_command.go:1430 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r166:/Table/84/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=119] I190903 18:32:32.733342 160533 storage/replica_raft.go:291 [n1,s1,r166/2:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:32.741848 160522 storage/store.go:2593 [n3,replicaGC,s3,r166/1:/Table/84/1/{4-5/6}] removing replica r166/1 I190903 18:32:32.742247 160522 storage/replica_destroy.go:152 [n3,replicaGC,s3,r166/1:/Table/84/1/{4-5/6}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:32.785884 160606 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r163:/Table/84/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=117] I190903 18:32:32.792558 160606 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:32.800603 160727 storage/store.go:2593 [n3,replicaGC,s3,r163/1:/Table/84/1{-/3}] removing replica r163/1 I190903 18:32:32.800931 160727 storage/replica_destroy.go:152 [n3,replicaGC,s3,r163/1:/Table/84/1{-/3}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:33.003768 161135 storage/replica_command.go:1430 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r168:/Table/84/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 18:32:33.009000 161135 storage/replica_raft.go:291 [n1,s1,r168/2:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:33.017780 161397 storage/store.go:2593 [n3,replicaGC,s3,r168/1:/Table/84/{1/5/7-2}] removing replica r168/1 I190903 18:32:33.018336 161397 storage/replica_destroy.go:152 [n3,replicaGC,s3,r168/1:/Table/84/{1/5/7-2}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:33.080914 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:33.084085 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:33.087395 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:33.090890 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:33.092788 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:33.103286 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=2 mtime=2019-09-03 18:32:33.096868876 +0000 UTC I190903 18:32:33.171757 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 1}] I190903 18:32:33.172733 161767 sql/lease.go:923 new lease: 84("multi col range partitioning - MAXVALUE MAXVALUE") ver=2:1567535862.669405797,0, refcount=0 I190903 18:32:33.230804 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=3 mtime=2019-09-03 18:32:33.229448788 +0000 UTC I190903 18:32:33.242007 162385 sql/lease.go:923 new lease: 84("multi col range partitioning - MAXVALUE MAXVALUE") ver=3:1567535863.179294656,0, refcount=0 I190903 18:32:33.277473 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=4 mtime=2019-09-03 18:32:33.258683381 +0000 UTC I190903 18:32:33.302339 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 84, info: {TableName:data.public."multi col range partitioning - MAXVALUE MAXVALUE" Statement:ALTER TABLE "multi col range partitioning - MAXVALUE MAXVALUE" PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p3x VALUES FROM (3, 4) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7), PARTITION pxx VALUES FROM (5, 7) TO (maxvalue, maxvalue)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:33.333833 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 3}] I190903 18:32:33.341860 162880 sql/lease.go:923 new lease: 84("multi col range partitioning - MAXVALUE MAXVALUE") ver=4:1567535863.180294656,0, refcount=0 I190903 18:32:33.472749 163813 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r163:/Table/84/1{-/3} [(n2,s2):2, next=3, gen=118] I190903 18:32:33.474982 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:33.477765 161036 storage/replica_command.go:284 [n3,split,s3,r165/1:/Table/84/1/{3-4}] initiating a split of this range at key /Table/84/1/3/4 [r169] (zone config) I190903 18:32:33.490212 163813 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/84/1{-/3}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:33.491365 163813 storage/store_snapshot.go:995 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] sending LEARNER snapshot 32f8f0a8 at applied index 38 I190903 18:32:33.491572 163813 storage/store_snapshot.go:1038 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:33.492824 164093 storage/replica_raftstorage.go:793 [n1,s1,r163/3:{-}] applying LEARNER snapshot [id=32f8f0a8 index=38] I190903 18:32:33.496828 164093 storage/replica_raftstorage.go:814 [n1,s1,r163/3:/Table/84/1{-/3}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=32f8f0a8 index=38] I190903 18:32:33.497294 163813 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] change replicas (add [(n1,s1):3] remove []): existing descriptor r163:/Table/84/1{-/3} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=119] I190903 18:32:33.498044 164228 storage/replica_command.go:1430 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r165:/Table/84/1/3{-/4} [(n3,s3):1, next=2, gen=117] I190903 18:32:33.510124 164228 storage/replica_raft.go:291 [n3,s3,r165/1:/Table/84/1/3{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:33.534117 164228 storage/store_snapshot.go:995 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] sending LEARNER snapshot 068505aa at applied index 27 I190903 18:32:33.534421 163813 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/84/1{-/3}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:33.534418 164228 storage/store_snapshot.go:1038 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 18:32:33.535006 163813 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r163:/Table/84/1{-/3} [(n2,s2):2, (n1,s1):3, next=4, gen=120] I190903 18:32:33.535648 163872 storage/replica_raftstorage.go:793 [n1,s1,r165/2:{-}] applying LEARNER snapshot [id=068505aa index=27] I190903 18:32:33.537027 163872 storage/replica_raftstorage.go:814 [n1,s1,r165/2:/Table/84/1/3{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=068505aa index=27] I190903 18:32:33.537543 164228 storage/replica_command.go:1430 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r165:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=118] I190903 18:32:33.623126 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:33.631651 164228 storage/replica_raft.go:291 [n3,s3,r165/1:/Table/84/1/3{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:33.632273 164228 storage/replica_command.go:1430 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r165:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=119] I190903 18:32:33.634910 163813 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:33.634946 163813 storage/replica_raft.go:304 [n2,s2,r163/2:/Table/84/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:33.649513 164778 storage/queue.go:1027 [n2,merge,s2,r163/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1,(n1,s1):2 I190903 18:32:33.655315 164228 storage/replica_raft.go:291 [n3,s3,r165/1:/Table/84/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:33.655365 164228 storage/replica_raft.go:304 [n3,s3,r165/1:/Table/84/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:33.655963 163813 storage/replica_command.go:979 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] could not promote [] to voter, rolling back: change replicas of r163 failed: storage/replica_raft.go:305: [n2,s2,r163/2:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:33.656011 163813 storage/queue.go:1027 [n2,replicate,s2,r163/2:/Table/84/1{-/3}] change replicas of r163 failed: storage/replica_raft.go:305: [n2,s2,r163/2:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:33.657299 164228 storage/replica_command.go:979 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r165 failed: storage/replica_raft.go:305: [n3,s3,r165/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:33.657373 164228 storage/queue.go:1027 [n3,replicate,s3,r165/1:/Table/84/1/3{-/4}] change replicas of r165 failed: storage/replica_raft.go:305: [n3,s3,r165/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:33.685672 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} E190903 18:32:33.699759 165380 storage/queue.go:1027 [n2,merge,s2,r163/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1,(n1,s1):2 I190903 18:32:33.753081 165574 storage/replica_command.go:1430 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r166:/Table/84/1/{4-5/6} [(n1,s1):2, next=3, gen=120] I190903 18:32:33.798655 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} E190903 18:32:33.804341 165924 storage/queue.go:1027 [n2,merge,s2,r163/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1,(n1,s1):2 I190903 18:32:33.818320 165574 storage/replica_raft.go:291 [n1,s1,r166/2:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:33.819652 165574 storage/store_snapshot.go:995 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] sending LEARNER snapshot 080b0abb at applied index 34 I190903 18:32:33.819947 165574 storage/store_snapshot.go:1038 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:33.822036 165523 storage/replica_raftstorage.go:793 [n3,s3,r166/3:{-}] applying LEARNER snapshot [id=080b0abb index=34] I190903 18:32:33.852585 165523 storage/replica_raftstorage.go:814 [n3,s3,r166/3:/Table/84/1/{4-5/6}] applied LEARNER snapshot [total=30ms ingestion=4@30ms id=080b0abb index=34] I190903 18:32:33.855136 165574 storage/replica_command.go:1430 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r166:/Table/84/1/{4-5/6} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=121] I190903 18:32:33.860070 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION pxx OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:33.860210 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:33.863077 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 E190903 18:32:33.864063 166427 storage/queue.go:1027 [n2,merge,s2,r163/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1,(n1,s1):2 I190903 18:32:33.867966 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:33.877017 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:33.888230 165574 storage/replica_raft.go:291 [n1,s1,r166/2:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 18:32:33.888788 165574 storage/replica_command.go:1430 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r166:/Table/84/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=122] I190903 18:32:33.895135 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:33.900768 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:33.903063 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:33.903657 165574 storage/replica_raft.go:291 [n1,s1,r166/2:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 18:32:33.903728 165574 storage/replica_raft.go:304 [n1,s1,r166/2:/Table/84/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:33.914663 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:33.920670 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:33.924079 165574 storage/replica_command.go:979 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r166 failed: storage/replica_raft.go:305: [n1,s1,r166/2:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:33.924110 165574 storage/queue.go:1027 [n1,replicate,s1,r166/2:/Table/84/1/{4-5/6}] change replicas of r166 failed: storage/replica_raft.go:305: [n1,s1,r166/2:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:33.928494 166213 storage/replica_command.go:1430 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r168:/Table/84/{1/5/7-2} [(n1,s1):2, next=3, gen=121] I190903 18:32:33.930129 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:33.952201 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:33.958840 166213 storage/replica_raft.go:291 [n1,s1,r168/2:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:33.963712 166933 storage/replica_command.go:1430 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r167:/Table/84/1/5/{6-7} [(n2,s2):2, next=3, gen=121] I190903 18:32:33.973376 166213 storage/store_snapshot.go:995 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] sending LEARNER snapshot 00a35df1 at applied index 29 I190903 18:32:33.973570 166213 storage/store_snapshot.go:1038 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 18:32:33.974751 166924 storage/replica_raftstorage.go:793 [n2,s2,r168/3:{-}] applying LEARNER snapshot [id=00a35df1 index=29] I190903 18:32:33.976747 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:33.980825 166933 storage/replica_raft.go:291 [n2,s2,r167/2:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:33.981415 166924 storage/replica_raftstorage.go:814 [n2,s2,r168/3:/Table/84/{1/5/7-2}] applied LEARNER snapshot [total=7ms ingestion=4@1ms id=00a35df1 index=29] I190903 18:32:33.982354 166213 storage/replica_command.go:1430 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r168:/Table/84/{1/5/7-2} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=122] I190903 18:32:33.983891 166933 storage/store_snapshot.go:995 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] sending LEARNER snapshot 77406a82 at applied index 37 I190903 18:32:33.984245 166933 storage/store_snapshot.go:1038 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:33.985574 167015 storage/replica_raftstorage.go:793 [n1,s1,r167/3:{-}] applying LEARNER snapshot [id=77406a82 index=37] I190903 18:32:33.987997 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:33.989799 166213 storage/replica_raft.go:291 [n1,s1,r168/2:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:33.990768 166213 storage/replica_command.go:1430 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r168:/Table/84/{1/5/7-2} [(n1,s1):2, (n2,s2):3, next=4, gen=123] I190903 18:32:34.001651 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:34.012224 166213 storage/replica_raft.go:291 [n1,s1,r168/2:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:34.012274 166213 storage/replica_raft.go:304 [n1,s1,r168/2:/Table/84/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:34.013121 166213 storage/replica_command.go:979 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r168 failed: storage/replica_raft.go:305: [n1,s1,r168/2:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:34.013168 166213 storage/queue.go:1027 [n1,replicate,s1,r168/2:/Table/84/{1/5/7-2}] change replicas of r168 failed: storage/replica_raft.go:305: [n1,s1,r168/2:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:34.015032 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:34.021443 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.029472 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:34.034187 167015 storage/replica_raftstorage.go:814 [n1,s1,r167/3:/Table/84/1/5/{6-7}] applied LEARNER snapshot [total=49ms ingestion=4@34ms id=77406a82 index=37] I190903 18:32:34.039195 166933 storage/replica_command.go:1430 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r167:/Table/84/1/5/{6-7} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=122] I190903 18:32:34.039517 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.056357 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.062231 166933 storage/replica_raft.go:291 [n2,s2,r167/2:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:34.062904 166933 storage/replica_command.go:1430 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r167:/Table/84/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=123] I190903 18:32:34.064874 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.075851 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:34.083599 166933 storage/replica_raft.go:291 [n2,s2,r167/2:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:34.083732 166933 storage/replica_raft.go:304 [n2,s2,r167/2:/Table/84/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:34.085575 166933 storage/replica_command.go:979 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r167 failed: storage/replica_raft.go:305: [n2,s2,r167/2:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:34.085614 166933 storage/queue.go:1027 [n2,replicate,s2,r167/2:/Table/84/1/5/{6-7}] change replicas of r167 failed: storage/replica_raft.go:305: [n2,s2,r167/2:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:34.085818 167392 storage/replica_command.go:1430 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r169:/Table/84/1/{3/4-4} [(n3,s3):1, next=2, gen=117] I190903 18:32:34.085986 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.094165 167392 storage/replica_raft.go:291 [n3,s3,r169/1:/Table/84/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:34.096390 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:34.107631 167392 storage/store_snapshot.go:995 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] sending LEARNER snapshot efc0ff91 at applied index 17 I190903 18:32:34.107781 167392 storage/store_snapshot.go:1038 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:34.107861 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.112988 167479 storage/replica_raftstorage.go:793 [n2,s2,r169/2:{-}] applying LEARNER snapshot [id=efc0ff91 index=17] I190903 18:32:34.126945 167479 storage/replica_raftstorage.go:814 [n2,s2,r169/2:/Table/84/1/{3/4-4}] applied LEARNER snapshot [total=14ms ingestion=4@2ms id=efc0ff91 index=17] I190903 18:32:34.127729 167392 storage/replica_command.go:1430 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r169:/Table/84/1/{3/4-4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=118] I190903 18:32:34.140595 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.150062 167392 storage/replica_raft.go:291 [n3,s3,r169/1:/Table/84/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:34.151071 167392 storage/replica_command.go:1430 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r169:/Table/84/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=119] I190903 18:32:34.178570 167392 storage/replica_raft.go:291 [n3,s3,r169/1:/Table/84/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:34.178616 167392 storage/replica_raft.go:304 [n3,s3,r169/1:/Table/84/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:34.180142 167392 storage/replica_command.go:979 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r169 failed: storage/replica_raft.go:305: [n3,s3,r169/1:/Table/84/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:34.180228 167392 storage/queue.go:1027 [n3,replicate,s3,r169/1:/Table/84/1/{3/4-4}] change replicas of r169 failed: storage/replica_raft.go:305: [n3,s3,r169/1:/Table/84/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:34.187374 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) E190903 18:32:34.208133 167876 storage/queue.go:1027 [n1,replicate,s1,r165/2:/Table/84/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:34] E190903 18:32:34.212305 167682 storage/queue.go:1027 [n1,replicate,s1,r163/3:/Table/84/1{-/3}] no removable replicas from range that needs a removal: [2:0, 3*:47] I190903 18:32:34.227149 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) E190903 18:32:34.254589 167667 storage/queue.go:1027 [n2,merge,s2,r163/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1,(n1,s1):2 I190903 18:32:34.258789 167895 storage/replica_command.go:1430 [n1,replicate,s1,r165/2:/Table/84/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r165:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=119] I190903 18:32:34.265784 167895 storage/replica_raft.go:291 [n1,s1,r165/2:/Table/84/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:34.274247 168037 storage/store.go:2593 [n3,replicaGC,s3,r165/1:/Table/84/1/3{-/4}] removing replica r165/1 I190903 18:32:34.274655 168037 storage/replica_destroy.go:152 [n3,replicaGC,s3,r165/1:/Table/84/1/3{-/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:34.292950 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 E190903 18:32:34.316750 168217 storage/queue.go:1027 [n1,merge,s1,r163/3:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):2 I190903 18:32:34.317237 168218 storage/replica_command.go:1430 [n1,replicate,s1,r163/3:/Table/84/1{-/3}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r163:/Table/84/1{-/3} [(n2,s2):2, (n1,s1):3, next=4, gen=120] I190903 18:32:34.323081 168218 storage/replica_raft.go:291 [n1,s1,r163/3:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:34.332951 168141 storage/store.go:2593 [n2,replicaGC,s2,r163/2:/Table/84/1{-/3}] removing replica r163/2 I190903 18:32:34.333357 168141 storage/replica_destroy.go:152 [n2,replicaGC,s2,r163/2:/Table/84/1{-/3}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:34.364565 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:34.367226 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:34.505533 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 18:32:34.522756 168251 storage/queue.go:1027 [n3,replicate,s3,r166/3:/Table/84/1/{4-5/6}] no removable replicas from range that needs a removal: [2:0, 3*:41] E190903 18:32:34.595725 168964 storage/queue.go:1027 [n1,replicate,s1,r167/3:/Table/84/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:44] E190903 18:32:34.645032 169030 storage/queue.go:1027 [n2,replicate,s2,r168/3:/Table/84/{1/5/7-2}] no removable replicas from range that needs a removal: [2:0, 3*:37] E190903 18:32:34.724481 169256 storage/queue.go:1027 [n2,replicate,s2,r169/2:/Table/84/1/{3/4-4}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:34.777638 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:34.783931 169483 storage/replica_command.go:1430 [n3,replicate,s3,r166/3:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r166:/Table/84/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=122] I190903 18:32:34.790703 169483 storage/replica_raft.go:291 [n3,s3,r166/3:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:34.799391 169495 storage/store.go:2593 [n1,replicaGC,s1,r166/2:/Table/84/1/{4-5/6}] removing replica r166/2 I190903 18:32:34.800130 169495 storage/replica_destroy.go:152 [n1,replicaGC,s1,r166/2:/Table/84/1/{4-5/6}] removed 8 (0+8) keys in 1ms [clear=0ms commit=0ms] I190903 18:32:35.050416 170204 storage/replica_command.go:609 [n1,merge,s1,r163/3:/Table/84/1{-/3}] initiating a merge of r165:/Table/84/1/3{-/4} [(n1,s1):2, next=3, gen=120] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:35.059678 176 storage/store.go:2593 [n1,s1,r163/3:/Table/84/1{-/3}] removing replica r165/2 I190903 18:32:35.101633 170377 storage/replica_command.go:1430 [n2,replicate,s2,r168/3:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r168:/Table/84/{1/5/7-2} [(n1,s1):2, (n2,s2):3, next=4, gen=123] I190903 18:32:35.110017 170377 storage/replica_raft.go:291 [n2,s2,r168/3:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:35.119689 170393 storage/store.go:2593 [n1,replicaGC,s1,r168/2:/Table/84/{1/5/7-2}] removing replica r168/2 I190903 18:32:35.120205 170393 storage/replica_destroy.go:152 [n1,replicaGC,s1,r168/2:/Table/84/{1/5/7-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:35.157501 170547 storage/replica_command.go:1430 [n1,replicate,s1,r167/3:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r167:/Table/84/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=123] I190903 18:32:35.164299 170547 storage/replica_raft.go:291 [n1,s1,r167/3:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:35.172843 170400 storage/store.go:2593 [n2,replicaGC,s2,r167/2:/Table/84/1/5/{6-7}] removing replica r167/2 I190903 18:32:35.173312 170400 storage/replica_destroy.go:152 [n2,replicaGC,s2,r167/2:/Table/84/1/5/{6-7}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:35.290948 170877 storage/replica_command.go:1430 [n2,replicate,s2,r169/2:/Table/84/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r169:/Table/84/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=119] I190903 18:32:35.298096 170877 storage/replica_raft.go:291 [n2,s2,r169/2:/Table/84/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:35.306123 170956 storage/store.go:2593 [n3,replicaGC,s3,r169/1:/Table/84/1/{3/4-4}] removing replica r169/1 I190903 18:32:35.306494 170956 storage/replica_destroy.go:152 [n3,replicaGC,s3,r169/1:/Table/84/1/{3/4-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:35.384204 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:35.387933 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:35.390149 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:35.396695 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:35.399959 152166 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) === RUN TestRepartitioning/multi_col_range_partitioning_-_MAXVALUE_MAXVALUE/multi_col_range_partitioning_-_MAXVALUE I190903 18:32:35.489574 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=5 mtime=2019-09-03 18:32:35.401890441 +0000 UTC I190903 18:32:35.511202 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 83, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning - MAXVALUE MAXVALUE"]} I190903 18:32:35.522484 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 4}] I190903 18:32:35.580100 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=6 mtime=2019-09-03 18:32:35.579468326 +0000 UTC I190903 18:32:35.634283 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 85, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:35.638070 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:35.653405 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 86, info: {TableName:data.public."multi col range partitioning - MAXVALUE MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p3x VALUES FROM (3, 4) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7), PARTITION pxx VALUES FROM (5, 7) TO (maxvalue, maxvalue)) User:root} I190903 18:32:35.678453 173238 storage/replica_command.go:284 [n3,split,s3,r164/1:/{Table/84/2-Max}] initiating a split of this range at key /Table/86 [r170] (zone config) I190903 18:32:35.760278 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:35.760822 173788 storage/replica_command.go:284 [n3,split,s3,r170/1:/{Table/86-Max}] initiating a split of this range at key /Table/86/1 [r171] (zone config) I190903 18:32:35.771712 173829 storage/replica_command.go:284 [n3,split,s3,r171/1:/{Table/86/1-Max}] initiating a split of this range at key /Table/86/1/3/4 [r172] (zone config) I190903 18:32:35.784953 173713 storage/replica_command.go:1430 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r171:/Table/86/1{-/3/4} [(n3,s3):1, next=2, gen=117] I190903 18:32:35.792311 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:35.792789 174250 storage/replica_command.go:284 [n3,split,s3,r172/1:/{Table/86/1/3…-Max}] initiating a split of this range at key /Table/86/1/4 [r173] (zone config) I190903 18:32:35.798471 173713 storage/replica_raft.go:291 [n3,s3,r171/1:/Table/86/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:35.802591 173713 storage/store_snapshot.go:995 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] sending LEARNER snapshot bf9a1efe at applied index 17 I190903 18:32:35.802751 173713 storage/store_snapshot.go:1038 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:35.804662 174080 storage/replica_raftstorage.go:793 [n1,s1,r171/2:{-}] applying LEARNER snapshot [id=bf9a1efe index=17] I190903 18:32:35.810040 174080 storage/replica_raftstorage.go:814 [n1,s1,r171/2:/Table/86/1{-/3/4}] applied LEARNER snapshot [total=5ms ingestion=4@4ms id=bf9a1efe index=17] I190903 18:32:35.810576 173713 storage/replica_command.go:1430 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r171:/Table/86/1{-/3/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=118] I190903 18:32:35.837013 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:35.841618 174083 storage/replica_command.go:284 [n3,split,s3,r173/1:/{Table/86/1/4-Max}] initiating a split of this range at key /Table/86/1/5/6 [r174] (zone config) I190903 18:32:35.843768 173713 storage/replica_raft.go:291 [n3,s3,r171/1:/Table/86/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:35.844870 173713 storage/replica_command.go:1430 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r171:/Table/86/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=119] I190903 18:32:35.861095 173713 storage/replica_raft.go:291 [n3,s3,r171/1:/Table/86/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:35.861241 173713 storage/replica_raft.go:304 [n3,s3,r171/1:/Table/86/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:35.864091 173713 storage/replica_command.go:979 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r171 failed: storage/replica_raft.go:305: [n3,s3,r171/1:/Table/86/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:35.864130 173713 storage/queue.go:1027 [n3,replicate,s3,r171/1:/Table/86/1{-/3/4}] change replicas of r171 failed: storage/replica_raft.go:305: [n3,s3,r171/1:/Table/86/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:35.865414 175121 storage/replica_command.go:1430 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r172:/Table/86/1/{3/4-4} [(n3,s3):1, next=2, gen=118] I190903 18:32:35.876380 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:35.879467 174960 storage/replica_command.go:284 [n3,split,s3,r174/1:/{Table/86/1/5…-Max}] initiating a split of this range at key /Table/86/1/5/7 [r175] (zone config) I190903 18:32:35.881027 175121 storage/replica_raft.go:291 [n3,s3,r172/1:/Table/86/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:35.884957 175121 storage/store_snapshot.go:995 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] sending LEARNER snapshot a7e1440a at applied index 17 I190903 18:32:35.885104 175121 storage/store_snapshot.go:1038 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:35.887173 175087 storage/replica_raftstorage.go:793 [n2,s2,r172/2:{-}] applying LEARNER snapshot [id=a7e1440a index=17] I190903 18:32:35.902092 175087 storage/replica_raftstorage.go:814 [n2,s2,r172/2:/Table/86/1/{3/4-4}] applied LEARNER snapshot [total=12ms ingestion=4@12ms id=a7e1440a index=17] I190903 18:32:35.903915 175121 storage/replica_command.go:1430 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r172:/Table/86/1/{3/4-4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=119] I190903 18:32:35.907961 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION pxx OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:35.908087 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:35.911486 175926 storage/replica_command.go:284 [n3,split,s3,r175/1:/{Table/86/1/5…-Max}] initiating a split of this range at key /Table/86/2 [r176] (zone config) I190903 18:32:35.915528 175121 storage/replica_raft.go:291 [n3,s3,r172/1:/Table/86/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:35.916104 175121 storage/replica_command.go:1430 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r172:/Table/86/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=120] I190903 18:32:35.937975 175121 storage/replica_raft.go:291 [n3,s3,r172/1:/Table/86/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:35.938010 175121 storage/replica_raft.go:304 [n3,s3,r172/1:/Table/86/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:35.945669 175121 storage/replica_command.go:979 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r172 failed: storage/replica_raft.go:305: [n3,s3,r172/1:/Table/86/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:35.945770 175121 storage/queue.go:1027 [n3,replicate,s3,r172/1:/Table/86/1/{3/4-4}] change replicas of r172 failed: storage/replica_raft.go:305: [n3,s3,r172/1:/Table/86/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:35.946444 175104 storage/replica_command.go:1430 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r174:/Table/86/1/5/{6-7} [(n3,s3):1, next=2, gen=120] I190903 18:32:35.951810 175104 storage/replica_raft.go:291 [n3,s3,r174/1:/Table/86/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:35.953230 175104 storage/store_snapshot.go:995 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] sending LEARNER snapshot 5928e463 at applied index 18 I190903 18:32:35.953379 175104 storage/store_snapshot.go:1038 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:35.954756 176247 storage/replica_raftstorage.go:793 [n1,s1,r174/2:{-}] applying LEARNER snapshot [id=5928e463 index=18] I190903 18:32:35.956148 176247 storage/replica_raftstorage.go:814 [n1,s1,r174/2:/Table/86/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5928e463 index=18] I190903 18:32:35.956190 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:35.961178 175104 storage/replica_command.go:1430 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] change replicas (add [(n1,s1):2] remove []): existing descriptor r174:/Table/86/1/5/{6-7} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=121] I190903 18:32:35.970291 175104 storage/replica_raft.go:291 [n3,s3,r174/1:/Table/86/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:35.970903 175104 storage/replica_command.go:1430 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r174:/Table/86/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=122] I190903 18:32:35.971132 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:35.988384 175104 storage/replica_raft.go:291 [n3,s3,r174/1:/Table/86/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:35.988429 175104 storage/replica_raft.go:304 [n3,s3,r174/1:/Table/86/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:35.990273 175104 storage/replica_command.go:979 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r174 failed: storage/replica_raft.go:305: [n3,s3,r174/1:/Table/86/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:35.990314 175104 storage/queue.go:1027 [n3,replicate,s3,r174/1:/Table/86/1/5/{6-7}] change replicas of r174 failed: storage/replica_raft.go:305: [n3,s3,r174/1:/Table/86/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:35.990789 176219 storage/replica_command.go:1430 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r54:/Table/6{4/2-6} [(n3,s3):1, next=2, gen=32] I190903 18:32:35.997011 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:35.997631 176219 storage/replica_raft.go:291 [n3,s3,r54/1:/Table/6{4/2-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:35.998843 176219 storage/store_snapshot.go:995 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] sending LEARNER snapshot 7dca4d97 at applied index 24 I190903 18:32:35.999007 176219 storage/store_snapshot.go:1038 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:36.001913 176469 storage/replica_raftstorage.go:793 [n2,s2,r54/2:{-}] applying LEARNER snapshot [id=7dca4d97 index=24] I190903 18:32:36.004198 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:36.007544 176469 storage/replica_raftstorage.go:814 [n2,s2,r54/2:/Table/6{4/2-6}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=7dca4d97 index=24] I190903 18:32:36.008427 176219 storage/replica_command.go:1430 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r54:/Table/6{4/2-6} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=33] I190903 18:32:36.011753 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.054641 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.058215 176219 storage/replica_raft.go:291 [n3,s3,r54/1:/Table/6{4/2-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:36.058902 176219 storage/replica_command.go:1430 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r54:/Table/6{4/2-6} [(n3,s3):1, (n2,s2):2, next=3, gen=34] I190903 18:32:36.061312 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:36.077060 176219 storage/replica_raft.go:291 [n3,s3,r54/1:/Table/6{4/2-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:36.077097 176219 storage/replica_raft.go:304 [n3,s3,r54/1:/Table/6{4/2-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:36.078006 176219 storage/replica_command.go:979 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] could not promote [] to voter, rolling back: change replicas of r54 failed: storage/replica_raft.go:305: [n3,s3,r54/1:/Table/6{4/2-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:36.078618 176219 storage/queue.go:1027 [n3,replicate,s3,r54/1:/Table/6{4/2-6}] change replicas of r54 failed: storage/replica_raft.go:305: [n3,s3,r54/1:/Table/6{4/2-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:36.078196 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:36.081246 176660 storage/replica_command.go:1430 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r175:/Table/86/{1/5/7-2} [(n3,s3):1, next=2, gen=121] I190903 18:32:36.095793 176660 storage/replica_raft.go:291 [n3,s3,r175/1:/Table/86/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:36.096410 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.097494 176660 storage/store_snapshot.go:995 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] sending LEARNER snapshot 3e031e21 at applied index 17 I190903 18:32:36.097950 176660 storage/store_snapshot.go:1038 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:36.100087 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.100107 176705 storage/replica_raftstorage.go:793 [n2,s2,r175/2:{-}] applying LEARNER snapshot [id=3e031e21 index=17] I190903 18:32:36.102499 176705 storage/replica_raftstorage.go:814 [n2,s2,r175/2:/Table/86/{1/5/7-2}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=3e031e21 index=17] I190903 18:32:36.103027 176660 storage/replica_command.go:1430 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r175:/Table/86/{1/5/7-2} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=122] I190903 18:32:36.110413 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:36.113819 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.114006 176660 storage/replica_raft.go:291 [n3,s3,r175/1:/Table/86/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:36.124782 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.125092 176660 storage/replica_command.go:1430 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r175:/Table/86/{1/5/7-2} [(n3,s3):1, (n2,s2):2, next=3, gen=123] I190903 18:32:36.129628 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:36.136525 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:36.147976 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.149914 176660 storage/replica_raft.go:291 [n3,s3,r175/1:/Table/86/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:36.149968 176660 storage/replica_raft.go:304 [n3,s3,r175/1:/Table/86/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:36.151971 176660 storage/replica_command.go:979 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r175 failed: storage/replica_raft.go:305: [n3,s3,r175/1:/Table/86/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:36.151998 176660 storage/queue.go:1027 [n3,replicate,s3,r175/1:/Table/86/{1/5/7-2}] change replicas of r175 failed: storage/replica_raft.go:305: [n3,s3,r175/1:/Table/86/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:36.153780 176980 storage/replica_command.go:1430 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r176:/{Table/86/2-Max} [(n3,s3):1, next=2, gen=121] E190903 18:32:36.154603 176983 storage/queue.go:1027 [n1,replicate,s1,r171/2:/Table/86/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:36.160087 176980 storage/replica_raft.go:291 [n3,s3,r176/1:/{Table/86/2-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:36.161227 176980 storage/store_snapshot.go:995 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] sending LEARNER snapshot 50f93055 at applied index 15 I190903 18:32:36.161846 176980 storage/store_snapshot.go:1038 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:36.162414 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.164174 177007 storage/replica_raftstorage.go:793 [n2,s2,r176/2:{-}] applying LEARNER snapshot [id=50f93055 index=15] I190903 18:32:36.165539 177007 storage/replica_raftstorage.go:814 [n2,s2,r176/2:/{Table/86/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=50f93055 index=15] I190903 18:32:36.169009 176980 storage/replica_command.go:1430 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r176:/{Table/86/2-Max} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=122] I190903 18:32:36.182889 176980 storage/replica_raft.go:291 [n3,s3,r176/1:/{Table/86/2-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:36.183555 176980 storage/replica_command.go:1430 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r176:/{Table/86/2-Max} [(n3,s3):1, (n2,s2):2, next=3, gen=123] I190903 18:32:36.207721 176980 storage/replica_raft.go:291 [n3,s3,r176/1:/{Table/86/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:36.207768 176980 storage/replica_raft.go:304 [n3,s3,r176/1:/{Table/86/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:36.208851 176980 storage/replica_command.go:979 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] could not promote [] to voter, rolling back: change replicas of r176 failed: storage/replica_raft.go:305: [n3,s3,r176/1:/{Table/86/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:36.208973 176980 storage/queue.go:1027 [n3,replicate,s3,r176/1:/{Table/86/2-Max}] change replicas of r176 failed: storage/replica_raft.go:305: [n3,s3,r176/1:/{Table/86/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:36.214737 177008 storage/replica_command.go:1430 [n1,replicate,s1,r171/2:/Table/86/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r171:/Table/86/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=119] I190903 18:32:36.222167 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.224635 177008 storage/replica_raft.go:291 [n1,s1,r171/2:/Table/86/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:36.226425 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.228928 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.233722 176974 storage/store.go:2593 [n3,replicaGC,s3,r171/1:/Table/86/1{-/3/4}] removing replica r171/1 I190903 18:32:36.235069 176974 storage/replica_destroy.go:152 [n3,replicaGC,s3,r171/1:/Table/86/1{-/3/4}] removed 9 (0+9) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:36.235096 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 18:32:36.243023 177375 storage/queue.go:1027 [n2,replicate,s2,r176/2:/{Table/86/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:36.243808 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:36.248863 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.250945 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.257877 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:36.263155 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:36.270736 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.275587 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.286733 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 18:32:36.298256 177582 storage/queue.go:1027 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:36.313808 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.331373 177690 storage/replica_command.go:1430 [n2,replicate,s2,r176/2:/{Table/86/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r176:/{Table/86/2-Max} [(n3,s3):1, (n2,s2):2, next=3, gen=123] I190903 18:32:36.342798 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.344123 177690 storage/replica_raft.go:291 [n2,s2,r176/2:/{Table/86/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:36.354659 177756 storage/store.go:2593 [n3,replicaGC,s3,r176/1:/{Table/86/2-Max}] removing replica r176/1 I190903 18:32:36.355047 177756 storage/replica_destroy.go:152 [n3,replicaGC,s3,r176/1:/{Table/86/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:36.382988 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.385222 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.399427 177725 storage/replica_command.go:1430 [n2,replicate,s2,r54/2:/Table/6{4/2-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r54:/Table/6{4/2-6} [(n3,s3):1, (n2,s2):2, next=3, gen=34] I190903 18:32:36.408397 177725 storage/replica_raft.go:291 [n2,s2,r54/2:/Table/6{4/2-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:36.432731 178003 storage/replica_command.go:1430 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r175:/Table/86/{1/5/7-2} [(n3,s3):1, (n2,s2):2, next=3, gen=123] I190903 18:32:36.442044 178003 storage/replica_raft.go:291 [n2,s2,r175/2:/Table/86/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:36.456296 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:36.461295 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.463289 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.477529 177482 storage/store.go:2593 [n3,replicaGC,s3,r54/1:/Table/6{4/2-6}] removing replica r54/1 I190903 18:32:36.478290 177482 storage/replica_destroy.go:152 [n3,replicaGC,s3,r54/1:/Table/6{4/2-6}] removed 8 (0+8) keys in 1ms [clear=0ms commit=0ms] I190903 18:32:36.479282 178212 storage/store.go:2593 [n3,replicaGC,s3,r175/1:/Table/86/{1/5/7-2}] removing replica r175/1 I190903 18:32:36.479616 178212 storage/replica_destroy.go:152 [n3,replicaGC,s3,r175/1:/Table/86/{1/5/7-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:36.518409 178358 storage/queue.go:1027 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:36.601133 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 E190903 18:32:36.735951 178922 storage/queue.go:1027 [n2,replicate,s2,r172/2:/Table/86/1/{3/4-4}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 18:32:36.795728 281 server/status/runtime.go:498 [n1] runtime stats: 269 MiB RSS, 972 goroutines, 77 MiB/23 MiB/138 MiB GO alloc/idle/total, 54 MiB/110 MiB CGO alloc/total, 8901.4 CGO/sec, 90.4/10.8 %(u/s)time, 0.8 %gc (13x), 98 MiB/98 MiB (r/w)net I190903 18:32:36.877680 179204 storage/replica_command.go:1430 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r174:/Table/86/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=122] I190903 18:32:36.883535 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:36.885175 179204 storage/replica_raft.go:291 [n1,s1,r174/2:/Table/86/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:36.888177 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 18:32:36.890427 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 18:32:36.893749 179358 storage/store.go:2593 [n3,replicaGC,s3,r174/1:/Table/86/1/5/{6-7}] removing replica r174/1 I190903 18:32:36.894052 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:36.894263 179358 storage/replica_destroy.go:152 [n3,replicaGC,s3,r174/1:/Table/86/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:36.901578 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:36.913700 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=86 (multi col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 18:32:36.905703212 +0000 UTC I190903 18:32:36.966647 179408 sql/lease.go:923 new lease: 86("multi col range partitioning - MAXVALUE") ver=2:1567535869.719303281,0, refcount=0 I190903 18:32:36.972954 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 86 1}] I190903 18:32:37.028565 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=86 (multi col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 18:32:37.027842134 +0000 UTC I190903 18:32:37.039138 180169 sql/lease.go:923 new lease: 86("multi col range partitioning - MAXVALUE") ver=3:1567535869.720303281,0, refcount=0 I190903 18:32:37.039426 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 86 2}] I190903 18:32:37.075501 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=86 (multi col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 18:32:37.066150506 +0000 UTC I190903 18:32:37.108131 180955 storage/replica_command.go:1430 [n1,s1,r171/2:/Table/86/1{-/3/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r171:/Table/86/1{-/3/4} [(n1,s1):2, next=3, gen=120] I190903 18:32:37.108455 180764 storage/replica_command.go:609 [n2,merge,s2,r175/2:/Table/86/{1/5/7-2}] initiating a merge of r176:/{Table/86/2-Max} [(n2,s2):2, next=3, gen=124] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:37.108866 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 86, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:ALTER TABLE "multi col range partitioning - MAXVALUE" PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:37.115557 180651 sql/lease.go:923 new lease: 86("multi col range partitioning - MAXVALUE") ver=4:1567535869.721303281,0, refcount=0 I190903 18:32:37.117569 180955 storage/replica_raft.go:291 [n1,s1,r171/2:/Table/86/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:37.118192 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 86 3}] I190903 18:32:37.119603 180955 storage/store_snapshot.go:995 [n1,s1,r171/2:/Table/86/1{-/3/4}] sending LEARNER snapshot ee962a37 at applied index 34 I190903 18:32:37.119961 180955 storage/store_snapshot.go:1038 [n1,s1,r171/2:/Table/86/1{-/3/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.125249 1529 storage/store.go:2593 [n2,s2,r175/2:/Table/86/{1/5/7-2}] removing replica r176/2 I190903 18:32:37.129115 181201 storage/replica_raftstorage.go:793 [n3,s3,r171/3:{-}] applying LEARNER snapshot [id=ee962a37 index=34] I190903 18:32:37.147384 181201 storage/replica_raftstorage.go:814 [n3,s3,r171/3:/Table/86/1{-/3/4}] applied LEARNER snapshot [total=18ms ingestion=4@18ms id=ee962a37 index=34] I190903 18:32:37.149289 180955 storage/replica_command.go:1430 [n1,s1,r171/2:/Table/86/1{-/3/4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r171:/Table/86/1{-/3/4} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=121] I190903 18:32:37.157161 180955 storage/replica_raft.go:291 [n1,s1,r171/2:/Table/86/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 E190903 18:32:37.160879 181421 storage/queue.go:1027 [n3,replicate,s3,r171/3:/Table/86/1{-/3/4}] no removable replicas from range that needs a removal: [2:0, 3*:38] I190903 18:32:37.161447 181444 storage/replica_command.go:1430 [n3,s3,r171/3:/Table/86/1{-/3/4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r171:/Table/86/1{-/3/4} [(n1,s1):2, (n3,s3):3, next=4, gen=122] I190903 18:32:37.171811 181444 storage/replica_raft.go:291 [n3,s3,r171/3:/Table/86/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:37.174246 180937 storage/replica_command.go:609 [n3,merge,s3,r170/1:/Table/86{-/1}] initiating a merge of r171:/Table/86/1{-/3/4} [(n3,s3):3, next=4, gen=123] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:37.180410 181406 storage/store.go:2593 [n1,replicaGC,s1,r171/2:/Table/86/1{-/3/4}] removing replica r171/2 I190903 18:32:37.181046 181406 storage/replica_destroy.go:152 [n1,replicaGC,s1,r171/2:/Table/86/1{-/3/4}] removed 8 (0+8) keys in 1ms [clear=0ms commit=0ms] I190903 18:32:37.181742 2322 storage/store.go:2593 [n3,s3,r170/1:/Table/86{-/1}] removing replica r171/3 I190903 18:32:37.189303 181560 storage/replica_command.go:1430 [n2,replicate,s2,r172/2:/Table/86/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r172:/Table/86/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=120] I190903 18:32:37.198869 181560 storage/replica_raft.go:291 [n2,s2,r172/2:/Table/86/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:37.201921 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:37.204025 181459 storage/replica_command.go:284 [n3,split,s3,r170/1:/Table/86{-/1/3/4}] initiating a split of this range at key /Table/86/1 [r177] (zone config) I190903 18:32:37.209443 181609 storage/store.go:2593 [n3,replicaGC,s3,r172/1:/Table/86/1/{3/4-4}] removing replica r172/1 I190903 18:32:37.209736 181609 storage/replica_destroy.go:152 [n3,replicaGC,s3,r172/1:/Table/86/1/{3/4-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:37.214547 182057 storage/replica_command.go:284 [n2,split,s2,r175/2:/{Table/86/1/5…-Max}] initiating a split of this range at key /Table/86/2 [r146] (zone config) I190903 18:32:37.231354 1651 server/status/runtime.go:498 [n2] runtime stats: 270 MiB RSS, 971 goroutines, 74 MiB/25 MiB/138 MiB GO alloc/idle/total, 53 MiB/111 MiB CGO alloc/total, 8647.9 CGO/sec, 87.4/10.6 %(u/s)time, 0.6 %gc (12x), 98 MiB/98 MiB (r/w)net I190903 18:32:37.237047 182096 storage/replica_command.go:1430 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r175:/Table/86/{1/5/7-2} [(n2,s2):2, next=3, gen=126] I190903 18:32:37.248355 182096 storage/replica_raft.go:291 [n2,s2,r175/2:/Table/86/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:37.250278 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:37.250638 182096 storage/store_snapshot.go:995 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] sending LEARNER snapshot 74b1132b at applied index 41 I190903 18:32:37.250964 182096 storage/store_snapshot.go:1038 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.252586 181587 storage/replica_raftstorage.go:793 [n1,s1,r175/3:{-}] applying LEARNER snapshot [id=74b1132b index=41] I190903 18:32:37.253643 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:37.255946 181587 storage/replica_raftstorage.go:814 [n1,s1,r175/3:/Table/86/{1/5/7-2}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=74b1132b index=41] I190903 18:32:37.256618 182096 storage/replica_command.go:1430 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] change replicas (add [(n1,s1):3] remove []): existing descriptor r175:/Table/86/{1/5/7-2} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=127] I190903 18:32:37.270381 182156 storage/replica_command.go:284 [n3,split,s3,r177/1:/Table/86/1{-/3/4}] initiating a split of this range at key /Table/86/1/3 [r178] (zone config) I190903 18:32:37.272020 182096 storage/replica_raft.go:291 [n2,s2,r175/2:/Table/86/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:37.275162 182096 storage/replica_command.go:1430 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r175:/Table/86/{1/5/7-2} [(n2,s2):2, (n1,s1):3, next=4, gen=128] I190903 18:32:37.283959 182558 storage/replica_command.go:1430 [n3,replicate,s3,r178/1:/Table/86/1/3{-/4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r178:/Table/86/1/3{-/4} [(n3,s3):1, next=2, gen=126] I190903 18:32:37.287123 182096 storage/replica_raft.go:291 [n2,s2,r175/2:/Table/86/{1/5/7-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:37.287181 182096 storage/replica_raft.go:304 [n2,s2,r175/2:/Table/86/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:37.287884 182583 storage/queue.go:1027 [n3,merge,s3,r178/1:/Table/86/1/3{-/4}] none of the remaining targets [(n3,s3):?] are legal additions to (n2,s2):2 I190903 18:32:37.288109 182096 storage/replica_command.go:979 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r175 failed: storage/replica_raft.go:305: [n2,s2,r175/2:/Table/86/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:37.288601 182096 storage/queue.go:1027 [n2,replicate,s2,r175/2:/Table/86/{1/5/7-2}] change replicas of r175 failed: storage/replica_raft.go:305: [n2,s2,r175/2:/Table/86/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:37.293003 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:37.297942 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:37.308799 182558 storage/replica_raft.go:291 [n3,s3,r178/1:/Table/86/1/3{-/4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:37.310718 182558 storage/store_snapshot.go:995 [n3,replicate,s3,r178/1:/Table/86/1/3{-/4}] sending LEARNER snapshot 48433d4b at applied index 15 I190903 18:32:37.310970 182558 storage/store_snapshot.go:1038 [n3,replicate,s3,r178/1:/Table/86/1/3{-/4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.312247 182932 storage/replica_command.go:1430 [n3,s3,r178/1:/Table/86/1/3{-/4}] change replicas (add [] remove [(n2,s2):2LEARNER]): existing descriptor r178:/Table/86/1/3{-/4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=127] I190903 18:32:37.312322 182699 storage/replica_raftstorage.go:793 [n2,s2,r178/2:{-}] applying LEARNER snapshot [id=48433d4b index=15] I190903 18:32:37.322852 182699 storage/replica_raftstorage.go:814 [n2,s2,r178/2:/Table/86/1/3{-/4}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=48433d4b index=15] I190903 18:32:37.323703 182932 storage/replica_raft.go:291 [n3,s3,r178/1:/Table/86/1/3{-/4}] proposing REMOVE_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1] next=3 I190903 18:32:37.324356 182558 storage/replica_command.go:979 [n3,replicate,s3,r178/1:/Table/86/1/3{-/4}] could not promote [n2,s2] to voter, rolling back: change replicas of r178 failed: descriptor changed: [expected] r178:/Table/86/1/3{-/4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=127] != [actual] r178:/Table/86/1/3{-/4} [(n3,s3):1, next=3, gen=128] I190903 18:32:37.326786 182723 storage/replica_command.go:1430 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r177:/Table/86/1{-/3} [(n3,s3):1, next=2, gen=126] I190903 18:32:37.326821 182934 storage/replica_command.go:1430 [n2,s2,r172/2:/Table/86/1/{3/4-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r172:/Table/86/1/{3/4-4} [(n2,s2):2, next=3, gen=121] I190903 18:32:37.334486 183039 storage/store.go:2593 [n2,replicaGC,s2,r178/2:/Table/86/1/3{-/4}] removing replica r178/2 I190903 18:32:37.334844 183039 storage/replica_destroy.go:152 [n2,replicaGC,s2,r178/2:/Table/86/1/3{-/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:37.344348 182934 storage/replica_raft.go:291 [n2,s2,r172/2:/Table/86/1/{3/4-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:37.344950 182723 storage/replica_raft.go:291 [n3,s3,r177/1:/Table/86/1{-/3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:37.346281 182723 storage/store_snapshot.go:995 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] sending LEARNER snapshot 55249a0c at applied index 19 I190903 18:32:37.346372 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:37.346448 182934 storage/store_snapshot.go:995 [n2,s2,r172/2:/Table/86/1/{3/4-4}] sending LEARNER snapshot e5c65f73 at applied index 33 I190903 18:32:37.346471 182723 storage/store_snapshot.go:1038 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 19, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.346618 182934 storage/store_snapshot.go:1038 [n2,s2,r172/2:/Table/86/1/{3/4-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.347975 183119 storage/replica_raftstorage.go:793 [n3,s3,r172/3:{-}] applying LEARNER snapshot [id=e5c65f73 index=33] I190903 18:32:37.348099 183323 storage/replica_raftstorage.go:793 [n2,s2,r177/2:{-}] applying LEARNER snapshot [id=55249a0c index=19] I190903 18:32:37.349551 183323 storage/replica_raftstorage.go:814 [n2,s2,r177/2:/Table/86/1{-/3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=55249a0c index=19] I190903 18:32:37.350113 182723 storage/replica_command.go:1430 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r177:/Table/86/1{-/3} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=127] I190903 18:32:37.356288 182723 storage/replica_raft.go:291 [n3,s3,r177/1:/Table/86/1{-/3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:37.356901 182723 storage/replica_command.go:1430 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r177:/Table/86/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=128] I190903 18:32:37.367885 182723 storage/replica_raft.go:291 [n3,s3,r177/1:/Table/86/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:37.367990 182723 storage/replica_raft.go:304 [n3,s3,r177/1:/Table/86/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:37.369916 182723 storage/replica_command.go:979 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] could not promote [] to voter, rolling back: change replicas of r177 failed: storage/replica_raft.go:305: [n3,s3,r177/1:/Table/86/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:37.369958 182723 storage/queue.go:1027 [n3,replicate,s3,r177/1:/Table/86/1{-/3}] change replicas of r177 failed: storage/replica_raft.go:305: [n3,s3,r177/1:/Table/86/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:37.372277 183119 storage/replica_raftstorage.go:814 [n3,s3,r172/3:/Table/86/1/{3/4-4}] applied LEARNER snapshot [total=24ms ingestion=4@22ms id=e5c65f73 index=33] I190903 18:32:37.372883 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 86, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:37.372943 182934 storage/replica_command.go:1430 [n2,s2,r172/2:/Table/86/1/{3/4-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r172:/Table/86/1/{3/4-4} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=122] I190903 18:32:37.373096 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.378010 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.386315 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.386765 182934 storage/replica_raft.go:291 [n2,s2,r172/2:/Table/86/1/{3/4-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 E190903 18:32:37.391216 183235 storage/queue.go:1027 [n3,replicate,s3,r172/3:/Table/86/1/{3/4-4}] no removable replicas from range that needs a removal: [2:0, 3*:37] I190903 18:32:37.391485 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:37.397544 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.402586 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.408151 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:37.415766 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:37.415835 184376 storage/replica_command.go:1430 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r173:/Table/86/1/{4-5/6} [(n3,s3):1, next=2, gen=119] I190903 18:32:37.421561 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.421774 184376 storage/replica_raft.go:291 [n3,s3,r173/1:/Table/86/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:37.423945 184376 storage/store_snapshot.go:995 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] sending LEARNER snapshot 09d4ec22 at applied index 23 I190903 18:32:37.424101 184376 storage/store_snapshot.go:1038 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.425266 184401 storage/replica_raftstorage.go:793 [n1,s1,r173/2:{-}] applying LEARNER snapshot [id=09d4ec22 index=23] I190903 18:32:37.425520 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.429780 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 18:32:37.434797 184401 storage/replica_raftstorage.go:814 [n1,s1,r173/2:/Table/86/1/{4-5/6}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=09d4ec22 index=23] I190903 18:32:37.436014 184376 storage/replica_command.go:1430 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r173:/Table/86/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=120] I190903 18:32:37.444573 184376 storage/replica_raft.go:291 [n3,s3,r173/1:/Table/86/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:37.445116 184376 storage/replica_command.go:1430 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r173:/Table/86/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=121] I190903 18:32:37.468443 184376 storage/replica_raft.go:291 [n3,s3,r173/1:/Table/86/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:37.468484 184376 storage/replica_raft.go:304 [n3,s3,r173/1:/Table/86/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:37.469440 184376 storage/replica_command.go:979 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r173 failed: storage/replica_raft.go:305: [n3,s3,r173/1:/Table/86/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:37.469470 184376 storage/queue.go:1027 [n3,replicate,s3,r173/1:/Table/86/1/{4-5/6}] change replicas of r173 failed: storage/replica_raft.go:305: [n3,s3,r173/1:/Table/86/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:37.483924 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:37.486041 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:37.488278 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.491026 184709 storage/replica_command.go:1430 [n3,s3,r172/3:/Table/86/1/{3/4-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r172:/Table/86/1/{3/4-4} [(n2,s2):2, (n3,s3):3, next=4, gen=123] I190903 18:32:37.496922 2361 server/status/runtime.go:498 [n3] runtime stats: 270 MiB RSS, 986 goroutines, 78 MiB/19 MiB/138 MiB GO alloc/idle/total, 53 MiB/110 MiB CGO alloc/total, 9213.8 CGO/sec, 93.5/11.0 %(u/s)time, 0.7 %gc (13x), 97 MiB/97 MiB (r/w)net I190903 18:32:37.498556 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.499780 184709 storage/replica_raft.go:291 [n3,s3,r172/3:/Table/86/1/{3/4-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:37.501985 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.502237 182932 storage/replica_command.go:609 [n3,merge,s3,r178/1:/Table/86/1/3{-/4}] initiating a merge of r172:/Table/86/1/{3/4-4} [(n3,s3):3, next=4, gen=124] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:37.505005 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.508399 184714 storage/store.go:2593 [n2,replicaGC,s2,r172/2:/Table/86/1/{3/4-4}] removing replica r172/2 I190903 18:32:37.508800 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:37.508920 184714 storage/replica_destroy.go:152 [n2,replicaGC,s2,r172/2:/Table/86/1/{3/4-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:37.509570 2264 storage/store.go:2593 [n3,s3,r178/1:/Table/86/1/3{-/4}] removing replica r172/3 I190903 18:32:37.510730 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.514106 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.518601 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.523494 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:37.526661 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.537996 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:37.551063 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:37.560719 184923 storage/replica_command.go:1430 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r174:/Table/86/1/5/{6-7} [(n1,s1):2, next=3, gen=123] I190903 18:32:37.564567 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.566070 184923 storage/replica_raft.go:291 [n1,s1,r174/2:/Table/86/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:37.567239 184923 storage/store_snapshot.go:995 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] sending LEARNER snapshot ab62692a at applied index 34 I190903 18:32:37.567501 184923 storage/store_snapshot.go:1038 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:37.569687 185030 storage/replica_raftstorage.go:793 [n2,s2,r174/3:{-}] applying LEARNER snapshot [id=ab62692a index=34] I190903 18:32:37.571777 185030 storage/replica_raftstorage.go:814 [n2,s2,r174/3:/Table/86/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ab62692a index=34] I190903 18:32:37.572257 184923 storage/replica_command.go:1430 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r174:/Table/86/1/5/{6-7} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=124] I190903 18:32:37.577500 184923 storage/replica_raft.go:291 [n1,s1,r174/2:/Table/86/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 18:32:37.578454 184923 storage/replica_command.go:1430 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r174:/Table/86/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=125] I190903 18:32:37.586668 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.590426 184923 storage/replica_raft.go:291 [n1,s1,r174/2:/Table/86/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 18:32:37.590460 184923 storage/replica_raft.go:304 [n1,s1,r174/2:/Table/86/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:37.591890 184923 storage/replica_command.go:979 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r174 failed: storage/replica_raft.go:305: [n1,s1,r174/2:/Table/86/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 18:32:37.591925 184923 storage/queue.go:1027 [n1,replicate,s1,r174/2:/Table/86/1/5/{6-7}] change replicas of r174 failed: storage/replica_raft.go:305: [n1,s1,r174/2:/Table/86/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 18:32:37.639717 185005 storage/replica_command.go:1430 [n1,replicate,s1,r175/3:/Table/86/{1/5/7-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r175:/Table/86/{1/5/7-2} [(n2,s2):2, (n1,s1):3, next=4, gen=128] I190903 18:32:37.644200 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:37.662673 185005 storage/replica_raft.go:291 [n1,s1,r175/3:/Table/86/{1/5/7-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:37.677808 185071 storage/store.go:2593 [n2,replicaGC,s2,r175/2:/Table/86/{1/5/7-2}] removing replica r175/2 I190903 18:32:37.678296 185071 storage/replica_destroy.go:152 [n2,replicaGC,s2,r175/2:/Table/86/{1/5/7-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:37.727435 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 18:32:37.827020 185790 storage/queue.go:1027 [n2,replicate,s2,r177/2:/Table/86/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:37.865204 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:38.047901 186406 storage/replica_command.go:1430 [n2,replicate,s2,r177/2:/Table/86/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r177:/Table/86/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=128] I190903 18:32:38.055986 186406 storage/replica_raft.go:291 [n2,s2,r177/2:/Table/86/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:38.065263 186440 storage/store.go:2593 [n3,replicaGC,s3,r177/1:/Table/86/1{-/3}] removing replica r177/1 I190903 18:32:38.065701 186440 storage/replica_destroy.go:152 [n3,replicaGC,s3,r177/1:/Table/86/1{-/3}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:38.154686 186734 storage/queue.go:1027 [n2,replicate,s2,r174/3:/Table/86/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:43] E190903 18:32:38.155596 186721 storage/queue.go:1027 [n2,replicate,s2,r174/3:/Table/86/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 18:32:38.189269 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 18:32:38.204371 186894 storage/queue.go:1027 [n1,replicate,s1,r173/2:/Table/86/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 18:32:38.484838 187296 storage/replica_command.go:1430 [n1,replicate,s1,r173/2:/Table/86/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r173:/Table/86/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=121] I190903 18:32:38.490699 187296 storage/replica_raft.go:291 [n1,s1,r173/2:/Table/86/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:38.506722 186475 storage/store.go:2593 [n3,replicaGC,s3,r173/1:/Table/86/1/{4-5/6}] removing replica r173/1 I190903 18:32:38.507136 186475 storage/replica_destroy.go:152 [n3,replicaGC,s3,r173/1:/Table/86/1/{4-5/6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:38.784189 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 18:32:38.786242 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 18:32:38.797129 188438 storage/replica_command.go:1430 [n2,replicate,s2,r174/3:/Table/86/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r174:/Table/86/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=125] I190903 18:32:38.804867 188438 storage/replica_raft.go:291 [n2,s2,r174/3:/Table/86/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:38.814086 188481 storage/store.go:2593 [n1,replicaGC,s1,r174/2:/Table/86/1/5/{6-7}] removing replica r174/2 I190903 18:32:38.814460 188481 storage/replica_destroy.go:152 [n1,replicaGC,s1,r174/2:/Table/86/1/5/{6-7}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:38.846204 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 18:32:38.848157 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 18:32:38.852790 171218 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 === RUN TestRepartitioning/single_col_list_partitioning/single_col_range_partitioning I190903 18:32:38.896756 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=86 (multi col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 18:32:38.85655028 +0000 UTC I190903 18:32:38.918648 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 85, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning - MAXVALUE"]} I190903 18:32:38.919675 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:38.927337 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 86 4}] I190903 18:32:38.984988 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=86 (multi col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 18:32:38.984159101 +0000 UTC I190903 18:32:39.033052 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 87, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:39.035058 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:39.045878 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 88, info: {TableName:data.public."single col list partitioning" Statement:CREATE TABLE "single col list partitioning" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p3 VALUES IN (3), PARTITION p4 VALUES IN (4)) User:root} I190903 18:32:39.049403 190679 storage/replica_command.go:284 [n2,split,s2,r146/1:/{Table/86/2-Max}] initiating a split of this range at key /Table/88 [r147] (zone config) I190903 18:32:39.087110 191082 storage/replica_command.go:284 [n2,split,s2,r147/1:/{Table/88-Max}] initiating a split of this range at key /Table/88/1 [r148] (zone config) I190903 18:32:39.087540 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 88, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:39.099324 191271 storage/replica_command.go:284 [n2,split,s2,r148/1:/{Table/88/1-Max}] initiating a split of this range at key /Table/88/2 [r149] (zone config) I190903 18:32:39.125957 191390 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/88/{1-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r148:/Table/88/{1-2} [(n2,s2):1, next=2, gen=129] I190903 18:32:39.128758 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 88, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:39.130120 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:39.131152 204 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:39.132219 191377 storage/replica_command.go:284 [n2,split,s2,r148/1:/Table/88/{1-2}] initiating a split of this range at key /Table/88/1/3 [r150] (zone config) I190903 18:32:39.133555 191390 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/88/{1-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:39.134678 191390 storage/store_snapshot.go:995 [n2,replicate,s2,r148/1:/Table/88/{1-2}] sending LEARNER snapshot 493f235a at applied index 17 I190903 18:32:39.134927 191390 storage/store_snapshot.go:1038 [n2,replicate,s2,r148/1:/Table/88/{1-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:39.136495 191774 storage/replica_raftstorage.go:793 [n1,s1,r148/2:{-}] applying LEARNER snapshot [id=493f235a index=17] I190903 18:32:39.140200 191377 storage/split_queue.go:149 [n2,split,s2,r148/1:/Table/88/{1-2}] split saw concurrent descriptor modification; maybe retrying I190903 18:32:39.140320 191650 storage/replica_command.go:284 [n2,split,s2,r148/1:/Table/88/{1-2}] initiating a split of this range at key /Table/88/1/3 [r151] (zone config); r148/2 is being probed (may or may not need a Raft snapshot) I190903 18:32:39.146070 191774 storage/replica_raftstorage.go:814 [n1,s1,r148/2:/Table/88/{1-2}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=493f235a index=17] I190903 18:32:39.165167 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 88, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:39.165326 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.165505 192117 storage/replica_command.go:284 [n2,split,s2,r151/1:/Table/88/{1/3-2}] initiating a split of this range at key /Table/88/1/4 [r152] (zone config); r151/2 is being probed (may or may not need a Raft snapshot) I190903 18:32:39.174304 191390 storage/replica_command.go:979 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] could not promote [n1,s1] to voter, rolling back: change replicas of r148 failed: descriptor changed: [expected] r148:/Table/88/{1-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=130] != [actual] r148:/Table/88/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=131] I190903 18:32:39.175504 191390 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] change replicas (add [] remove [(n1,s1):2LEARNER]): existing descriptor r148:/Table/88/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=131] I190903 18:32:39.185771 192390 storage/replica_command.go:284 [n2,split,s2,r152/1:/Table/88/{1/4-2}] initiating a split of this range at key /Table/88/1/5 [r153] (zone config); r152/2 is being probed (may or may not need a Raft snapshot) I190903 18:32:39.186580 191390 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/88/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1] next=3 I190903 18:32:39.187286 191390 storage/replica_command.go:1217 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] rolled back learner n1,s1 in r148:/Table/88/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=131] I190903 18:32:39.187806 192339 storage/replica_command.go:1430 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] change replicas (add [] remove [(n1,s1):2LEARNER]): existing descriptor r151:/Table/88/1/{3-4} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=132] I190903 18:32:39.193009 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.196461 192374 storage/store.go:2593 [n1,replicaGC,s1,r148/2:/Table/88/1{-/3}] removing replica r148/2 I190903 18:32:39.196514 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.196755 192374 storage/replica_destroy.go:152 [n1,replicaGC,s1,r148/2:/Table/88/1{-/3}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:39.197892 192339 storage/replica_raft.go:291 [n2,s2,r151/1:/Table/88/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1] next=3 I190903 18:32:39.198768 192382 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r148:/Table/88/1{-/3} [(n2,s2):1, next=3, gen=132] I190903 18:32:39.208769 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:39.212006 192225 storage/store.go:2593 [n1,replicaGC,s1,r151/2:/Table/88/1/{3-4}] removing replica r151/2 I190903 18:32:39.212356 192225 storage/replica_destroy.go:152 [n1,replicaGC,s1,r151/2:/Table/88/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:39.218996 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.222790 192382 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/88/1{-/3}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):1 (n1,s1):3LEARNER] next=4 I190903 18:32:39.224806 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.225447 192382 storage/store_snapshot.go:995 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] sending LEARNER snapshot 8e9e63cb at applied index 25 I190903 18:32:39.225672 192382 storage/store_snapshot.go:1038 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:39.228449 192605 storage/replica_raftstorage.go:793 [n1,s1,r148/3:{-}] applying LEARNER snapshot [id=8e9e63cb index=25] I190903 18:32:39.229330 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.235092 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.238371 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.244575 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.249012 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.256305 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.260902 192605 storage/replica_raftstorage.go:814 [n1,s1,r148/3:/Table/88/1{-/3}] applied LEARNER snapshot [total=32ms ingestion=4@32ms id=8e9e63cb index=25] I190903 18:32:39.263710 192382 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] change replicas (add [(n1,s1):3] remove []): existing descriptor r148:/Table/88/1{-/3} [(n2,s2):1, (n1,s1):3LEARNER, next=4, gen=133] I190903 18:32:39.265694 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:39.268876 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.272406 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.281157 192382 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/88/1{-/3}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):1 (n1,s1):3] next=4 I190903 18:32:39.283424 192382 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r148:/Table/88/1{-/3} [(n2,s2):1, (n1,s1):3, next=4, gen=134] I190903 18:32:39.283518 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:39.288048 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.298052 192382 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/88/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 E190903 18:32:39.298121 192382 storage/replica_raft.go:304 [n2,s2,r148/1:/Table/88/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:39.299200 192382 storage/replica_command.go:979 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] could not promote [] to voter, rolling back: change replicas of r148 failed: storage/replica_raft.go:305: [n2,s2,r148/1:/Table/88/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:39.299268 192382 storage/queue.go:1027 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] change replicas of r148 failed: storage/replica_raft.go:305: [n2,s2,r148/1:/Table/88/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:39.299763 192954 storage/replica_command.go:1430 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] change replicas (add [] remove [(n1,s1):2LEARNER]): existing descriptor r152:/Table/88/1/{4-5} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=133] I190903 18:32:39.301547 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.313829 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.317516 192954 storage/replica_raft.go:291 [n2,s2,r152/1:/Table/88/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1] next=3 I190903 18:32:39.317821 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.319497 192954 storage/replica_command.go:1430 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r152:/Table/88/1/{4-5} [(n2,s2):1, next=3, gen=134] I190903 18:32:39.326250 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.333946 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.334555 192954 storage/replica_raft.go:291 [n2,s2,r152/1:/Table/88/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):1 (n3,s3):3LEARNER] next=4 I190903 18:32:39.336774 192954 storage/store_snapshot.go:995 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] sending LEARNER snapshot 536d19c3 at applied index 20 I190903 18:32:39.336986 192954 storage/store_snapshot.go:1038 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:39.338268 192908 storage/replica_raftstorage.go:793 [n3,s3,r152/3:{-}] applying LEARNER snapshot [id=536d19c3 index=20] I190903 18:32:39.339656 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:39.343104 193019 storage/store.go:2593 [n1,replicaGC,s1,r152/2:/Table/88/1/{4-5}] removing replica r152/2 I190903 18:32:39.343504 193019 storage/replica_destroy.go:152 [n1,replicaGC,s1,r152/2:/Table/88/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:39.346981 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.347078 192908 storage/replica_raftstorage.go:814 [n3,s3,r152/3:/Table/88/1/{4-5}] applied LEARNER snapshot [total=9ms ingestion=4@7ms id=536d19c3 index=20] I190903 18:32:39.347569 192954 storage/replica_command.go:1430 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] change replicas (add [(n3,s3):3] remove []): existing descriptor r152:/Table/88/1/{4-5} [(n2,s2):1, (n3,s3):3LEARNER, next=4, gen=135] I190903 18:32:39.352199 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:39.357840 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.359065 192954 storage/replica_raft.go:291 [n2,s2,r152/1:/Table/88/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):1 (n3,s3):3] next=4 I190903 18:32:39.360019 192954 storage/replica_command.go:1430 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r152:/Table/88/1/{4-5} [(n2,s2):1, (n3,s3):3, next=4, gen=136] I190903 18:32:39.364649 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:39.371327 192954 storage/replica_raft.go:291 [n2,s2,r152/1:/Table/88/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 E190903 18:32:39.371365 192954 storage/replica_raft.go:304 [n2,s2,r152/1:/Table/88/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:39.372186 192954 storage/replica_command.go:979 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r152 failed: storage/replica_raft.go:305: [n2,s2,r152/1:/Table/88/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:39.372225 192954 storage/queue.go:1027 [n2,replicate,s2,r152/1:/Table/88/1/{4-5}] change replicas of r152 failed: storage/replica_raft.go:305: [n2,s2,r152/1:/Table/88/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:39.372607 193320 storage/replica_command.go:1430 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] change replicas (add [] remove [(n1,s1):2LEARNER]): existing descriptor r153:/Table/88/{1/5-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=133] I190903 18:32:39.374298 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.399877 193320 storage/replica_raft.go:291 [n2,s2,r153/1:/Table/88/{1/5-2}] proposing REMOVE_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1] next=3 I190903 18:32:39.400581 193320 storage/replica_command.go:1430 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r153:/Table/88/{1/5-2} [(n2,s2):1, next=3, gen=134] I190903 18:32:39.405530 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.408061 193070 storage/store.go:2593 [n1,replicaGC,s1,r153/2:/Table/88/{1/5-2}] removing replica r153/2 I190903 18:32:39.408651 193070 storage/replica_destroy.go:152 [n1,replicaGC,s1,r153/2:/Table/88/{1/5-2}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:39.422908 193320 storage/replica_raft.go:291 [n2,s2,r153/1:/Table/88/{1/5-2}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):1 (n1,s1):3LEARNER] next=4 I190903 18:32:39.424281 193320 storage/store_snapshot.go:995 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] sending LEARNER snapshot 7bd413e6 at applied index 17 I190903 18:32:39.424741 193320 storage/store_snapshot.go:1038 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:39.426601 193546 storage/replica_raftstorage.go:793 [n1,s1,r153/3:{-}] applying LEARNER snapshot [id=7bd413e6 index=17] I190903 18:32:39.428263 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.431899 193546 storage/replica_raftstorage.go:814 [n1,s1,r153/3:/Table/88/{1/5-2}] applied LEARNER snapshot [total=5ms ingestion=4@5ms id=7bd413e6 index=17] I190903 18:32:39.432513 193320 storage/replica_command.go:1430 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] change replicas (add [(n1,s1):3] remove []): existing descriptor r153:/Table/88/{1/5-2} [(n2,s2):1, (n1,s1):3LEARNER, next=4, gen=135] I190903 18:32:39.443005 193320 storage/replica_raft.go:291 [n2,s2,r153/1:/Table/88/{1/5-2}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):1 (n1,s1):3] next=4 I190903 18:32:39.469195 193320 storage/replica_command.go:1430 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r153:/Table/88/{1/5-2} [(n2,s2):1, (n1,s1):3, next=4, gen=136] I190903 18:32:39.471379 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:39.542531 193320 storage/replica_raft.go:291 [n2,s2,r153/1:/Table/88/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 E190903 18:32:39.542618 193320 storage/replica_raft.go:304 [n2,s2,r153/1:/Table/88/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:39.543492 193320 storage/replica_command.go:979 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r153 failed: storage/replica_raft.go:305: [n2,s2,r153/1:/Table/88/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:39.543576 193320 storage/queue.go:1027 [n2,replicate,s2,r153/1:/Table/88/{1/5-2}] change replicas of r153 failed: storage/replica_raft.go:305: [n2,s2,r153/1:/Table/88/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:39.577898 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:39.722592 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:39.782497 194533 storage/queue.go:1027 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 3*:27] I190903 18:32:39.912502 194846 storage/replica_command.go:1430 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r152:/Table/88/1/{4-5} [(n2,s2):1, (n3,s3):3, next=4, gen=136] I190903 18:32:39.919077 194846 storage/replica_raft.go:291 [n3,s3,r152/3:/Table/88/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 I190903 18:32:39.930562 194889 storage/store.go:2593 [n2,replicaGC,s2,r152/1:/Table/88/1/{4-5}] removing replica r152/1 I190903 18:32:39.931044 194889 storage/replica_destroy.go:152 [n2,replicaGC,s2,r152/1:/Table/88/1/{4-5}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:39.939901 194893 storage/queue.go:1027 [n1,replicate,s1,r148/3:/Table/88/1{-/3}] no removable replicas from range that needs a removal: [1:0, 3*:33] I190903 18:32:39.995379 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:40.000820 194946 storage/queue.go:1027 [n1,replicate,s1,r153/3:/Table/88/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 3*:24] I190903 18:32:40.002410 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:40.006919 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:40.011226 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:40.021259 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=88 (single col range partitioning) version=2 mtime=2019-09-03 18:32:40.015636116 +0000 UTC I190903 18:32:40.049331 195271 sql/lease.go:923 new lease: 88("single col range partitioning") ver=2:1567535869.911380527,0, refcount=0 I190903 18:32:40.060736 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 88 1}] I190903 18:32:40.071254 195743 storage/replica_command.go:1430 [n1,replicate,s1,r153/3:/Table/88/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r153:/Table/88/{1/5-2} [(n2,s2):1, (n1,s1):3, next=4, gen=136] I190903 18:32:40.086242 195743 storage/replica_raft.go:291 [n1,s1,r153/3:/Table/88/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):3] next=4 I190903 18:32:40.105939 195862 storage/store.go:2593 [n2,replicaGC,s2,r153/1:/Table/88/{1/5-2}] removing replica r153/1 I190903 18:32:40.106754 195862 storage/replica_destroy.go:152 [n2,replicaGC,s2,r153/1:/Table/88/{1/5-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:40.127135 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=88 (single col range partitioning) version=3 mtime=2019-09-03 18:32:40.126333793 +0000 UTC I190903 18:32:40.177323 196057 sql/lease.go:923 new lease: 88("single col range partitioning") ver=3:1567535869.912380527,0, refcount=0 I190903 18:32:40.181606 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 88 2}] I190903 18:32:40.219671 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=88 (single col range partitioning) version=4 mtime=2019-09-03 18:32:40.215502893 +0000 UTC I190903 18:32:40.236545 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 88, info: {TableName:data.public."single col range partitioning" Statement:ALTER TABLE "single col range partitioning" PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:40.242644 196537 sql/lease.go:923 new lease: 88("single col range partitioning") ver=4:1567535869.913380527,0, refcount=0 I190903 18:32:40.247115 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 88 3}] I190903 18:32:40.312579 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 88, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} E190903 18:32:40.318996 197668 storage/queue.go:1027 [n3,merge,s3,r152/3:/Table/88/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):3 I190903 18:32:40.345156 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 88, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} E190903 18:32:40.351641 198028 storage/queue.go:1027 [n3,merge,s3,r152/3:/Table/88/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):3 I190903 18:32:40.373453 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 88, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:40.373602 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 18:32:40.378945 198365 storage/queue.go:1027 [n3,merge,s3,r152/3:/Table/88/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):3 I190903 18:32:40.380988 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:40.392782 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.396893 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.401480 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.403144 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.405007 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:40.408496 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:40.411489 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.413002 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:40.415978 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.417442 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.419116 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.420668 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:40.423733 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.425777 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.427788 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.429768 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.431549 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:40.434543 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:40.437833 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.440851 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.445478 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.459136 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.470056 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.488893 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.524411 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:40.595048 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:40.731141 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 18:32:40.862609 199933 storage/queue.go:1027 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] no removable replicas from range that needs a removal: [1*:39, 3:0] I190903 18:32:40.890258 200122 storage/replica_command.go:1430 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] change replicas (add [(n1,s1):4LEARNER] remove []): existing descriptor r152:/Table/88/1/{4-5} [(n3,s3):3, next=4, gen=137] E190903 18:32:40.891724 200121 storage/queue.go:1027 [n3,merge,s3,r152/3:/Table/88/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):3 I190903 18:32:40.897875 200122 storage/replica_raft.go:291 [n3,s3,r152/3:/Table/88/1/{4-5}] proposing ADD_REPLICA[(n1,s1):4LEARNER]: after=[(n3,s3):3 (n1,s1):4LEARNER] next=5 I190903 18:32:40.899437 200122 storage/store_snapshot.go:995 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] sending LEARNER snapshot 95169af0 at applied index 37 I190903 18:32:40.899769 200122 storage/store_snapshot.go:1038 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] streamed snapshot to (n1,s1):4LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:40.900823 200126 storage/replica_raftstorage.go:793 [n1,s1,r152/4:{-}] applying LEARNER snapshot [id=95169af0 index=37] I190903 18:32:40.902190 200126 storage/replica_raftstorage.go:814 [n1,s1,r152/4:/Table/88/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=95169af0 index=37] I190903 18:32:40.902803 200122 storage/replica_command.go:1430 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] change replicas (add [(n1,s1):4] remove []): existing descriptor r152:/Table/88/1/{4-5} [(n3,s3):3, (n1,s1):4LEARNER, next=5, gen=138] I190903 18:32:40.909219 200122 storage/replica_raft.go:291 [n3,s3,r152/3:/Table/88/1/{4-5}] proposing ADD_REPLICA[(n1,s1):4]: after=[(n3,s3):3 (n1,s1):4] next=5 I190903 18:32:40.909920 200122 storage/replica_command.go:1430 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r152:/Table/88/1/{4-5} [(n3,s3):3, (n1,s1):4, next=5, gen=139] I190903 18:32:40.924501 200122 storage/replica_raft.go:291 [n3,s3,r152/3:/Table/88/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):4] next=5 E190903 18:32:40.924587 200122 storage/replica_raft.go:304 [n3,s3,r152/3:/Table/88/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):4] next=5 to remove self (leaseholder) I190903 18:32:40.925764 200122 storage/replica_command.go:979 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r152 failed: storage/replica_raft.go:305: [n3,s3,r152/3:/Table/88/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):4] next=5 to remove self (leaseholder) E190903 18:32:40.925800 200122 storage/queue.go:1027 [n3,replicate,s3,r152/3:/Table/88/1/{4-5}] change replicas of r152 failed: storage/replica_raft.go:305: [n3,s3,r152/3:/Table/88/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):4] next=5 to remove self (leaseholder) I190903 18:32:41.002807 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:41.090196 200694 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/88/1{-/3}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r148:/Table/88/1{-/3} [(n2,s2):1, (n1,s1):3, next=4, gen=134] I190903 18:32:41.096711 200694 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/88/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):1] next=4 I190903 18:32:41.107248 198793 storage/store.go:2593 [n1,replicaGC,s1,r148/3:/Table/88/1{-/3}] removing replica r148/3 I190903 18:32:41.107661 198793 storage/replica_destroy.go:152 [n1,replicaGC,s1,r148/3:/Table/88/1{-/3}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:41.252298 201124 storage/replica_command.go:1430 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r151:/Table/88/1/{3-4} [(n2,s2):1, next=3, gen=133] I190903 18:32:41.258834 201124 storage/replica_raft.go:291 [n2,s2,r151/1:/Table/88/1/{3-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):1 (n3,s3):3LEARNER] next=4 I190903 18:32:41.259980 201124 storage/store_snapshot.go:995 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] sending LEARNER snapshot cc131481 at applied index 25 I190903 18:32:41.260169 201124 storage/store_snapshot.go:1038 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:41.261471 200681 storage/replica_raftstorage.go:793 [n3,s3,r151/3:{-}] applying LEARNER snapshot [id=cc131481 index=25] I190903 18:32:41.267790 200681 storage/replica_raftstorage.go:814 [n3,s3,r151/3:/Table/88/1/{3-4}] applied LEARNER snapshot [total=6ms ingestion=4@5ms id=cc131481 index=25] I190903 18:32:41.268267 201124 storage/replica_command.go:1430 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r151:/Table/88/1/{3-4} [(n2,s2):1, (n3,s3):3LEARNER, next=4, gen=134] I190903 18:32:41.275557 201124 storage/replica_raft.go:291 [n2,s2,r151/1:/Table/88/1/{3-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):1 (n3,s3):3] next=4 I190903 18:32:41.276168 201124 storage/replica_command.go:1430 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r151:/Table/88/1/{3-4} [(n2,s2):1, (n3,s3):3, next=4, gen=135] I190903 18:32:41.291001 201124 storage/replica_raft.go:291 [n2,s2,r151/1:/Table/88/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 E190903 18:32:41.291789 201124 storage/replica_raft.go:304 [n2,s2,r151/1:/Table/88/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:41.294303 201124 storage/replica_command.go:979 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r151 failed: storage/replica_raft.go:305: [n2,s2,r151/1:/Table/88/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:41.294409 201124 storage/queue.go:1027 [n2,replicate,s2,r151/1:/Table/88/1/{3-4}] change replicas of r151 failed: storage/replica_raft.go:305: [n2,s2,r151/1:/Table/88/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:41.519205 201865 storage/queue.go:1027 [n3,merge,s3,r152/3:/Table/88/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):3 E190903 18:32:41.519399 201879 storage/queue.go:1027 [n1,replicate,s1,r152/4:/Table/88/1/{4-5}] no removable replicas from range that needs a removal: [3:0, 4*:45] I190903 18:32:41.544688 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:41.548323 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 18:32:41.727721 202468 storage/queue.go:1027 [n1,merge,s1,r152/4:/Table/88/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):3 I190903 18:32:41.775446 202469 storage/replica_command.go:1430 [n1,replicate,s1,r152/4:/Table/88/1/{4-5}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r152:/Table/88/1/{4-5} [(n3,s3):3, (n1,s1):4, next=5, gen=139] I190903 18:32:41.783099 202469 storage/replica_raft.go:291 [n1,s1,r152/4:/Table/88/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):4] next=5 I190903 18:32:41.792365 202599 storage/store.go:2593 [n3,replicaGC,s3,r152/3:/Table/88/1/{4-5}] removing replica r152/3 I190903 18:32:41.792810 202599 storage/replica_destroy.go:152 [n3,replicaGC,s3,r152/3:/Table/88/1/{4-5}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] E190903 18:32:41.912228 202913 storage/queue.go:1027 [n3,replicate,s3,r151/3:/Table/88/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 3*:32] I190903 18:32:42.552292 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:42.555746 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:42.560234 188571 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 === RUN TestRepartitioning/single_col_range_partitioning/single_col_list_partitioning I190903 18:32:42.594523 204776 storage/replica_command.go:609 [n1,merge,s1,r152/4:/Table/88/1/{4-5}] initiating a merge of r153:/Table/88/{1/5-2} [(n1,s1):3, next=4, gen=137] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:42.618369 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=88 (single col range partitioning) version=5 mtime=2019-09-03 18:32:42.563628998 +0000 UTC I190903 18:32:42.668043 218 storage/store.go:2593 [n1,s1,r152/4:/Table/88/1/{4-5}] removing replica r153/3 I190903 18:32:42.680879 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 87, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning"]} I190903 18:32:42.709454 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 88 4}] I190903 18:32:42.772000 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=88 (single col range partitioning) version=6 mtime=2019-09-03 18:32:42.771084233 +0000 UTC I190903 18:32:42.791944 205588 storage/replica_command.go:1430 [n3,replicate,s3,r151/3:/Table/88/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r151:/Table/88/1/{3-4} [(n2,s2):1, (n3,s3):3, next=4, gen=135] I190903 18:32:42.797647 205588 storage/replica_raft.go:291 [n3,s3,r151/3:/Table/88/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 I190903 18:32:42.806589 205644 storage/store.go:2593 [n2,replicaGC,s2,r151/1:/Table/88/1/{3-4}] removing replica r151/1 I190903 18:32:42.806992 205644 storage/replica_destroy.go:152 [n2,replicaGC,s2,r151/1:/Table/88/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:42.818879 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 89, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:42.844137 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 90, info: {TableName:data.public."single col range partitioning" Statement:CREATE TABLE "single col range partitioning" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root} I190903 18:32:42.845524 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:42.849077 207067 storage/replica_command.go:284 [n2,split,s2,r149/1:/{Table/88/2-Max}] initiating a split of this range at key /Table/90 [r154] (zone config) I190903 18:32:42.876032 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 90, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:42.882828 206943 storage/replica_command.go:284 [n2,split,s2,r154/1:/{Table/90-Max}] initiating a split of this range at key /Table/90/1 [r155] (zone config) I190903 18:32:42.892461 206819 storage/replica_command.go:284 [n2,split,s2,r155/1:/{Table/90/1-Max}] initiating a split of this range at key /Table/90/2 [r156] (zone config) I190903 18:32:42.909655 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 90, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:42.918489 208072 storage/replica_command.go:284 [n2,split,s2,r155/1:/Table/90/{1-2}] initiating a split of this range at key /Table/90/1/3 [r157] (zone config) I190903 18:32:42.919211 206960 storage/replica_command.go:1430 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r156:/{Table/90/2-Max} [(n2,s2):1, next=2, gen=132] I190903 18:32:42.953656 206960 storage/replica_raft.go:291 [n2,s2,r156/1:/{Table/90/2-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:42.955433 206960 storage/store_snapshot.go:995 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] sending LEARNER snapshot 27a11ba8 at applied index 14 I190903 18:32:42.955604 206960 storage/store_snapshot.go:1038 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:42.958185 207793 storage/replica_raftstorage.go:793 [n3,s3,r156/2:{-}] applying LEARNER snapshot [id=27a11ba8 index=14] I190903 18:32:42.959535 207793 storage/replica_raftstorage.go:814 [n3,s3,r156/2:/{Table/90/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=27a11ba8 index=14] I190903 18:32:42.960079 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 90, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:42.962890 208436 storage/replica_command.go:284 [n2,split,s2,r157/1:/Table/90/{1/3-2}] initiating a split of this range at key /Table/90/1/4 [r158] (zone config) I190903 18:32:42.978479 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:42.981715 206960 storage/replica_command.go:1430 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r156:/{Table/90/2-Max} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=133] I190903 18:32:43.011771 206960 storage/replica_raft.go:291 [n2,s2,r156/1:/{Table/90/2-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 18:32:43.012331 206960 storage/replica_command.go:1430 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r156:/{Table/90/2-Max} [(n2,s2):1, (n3,s3):2, next=3, gen=134] I190903 18:32:43.033105 206960 storage/replica_raft.go:291 [n2,s2,r156/1:/{Table/90/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 18:32:43.033157 206960 storage/replica_raft.go:304 [n2,s2,r156/1:/{Table/90/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:43.034392 206960 storage/replica_command.go:979 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] could not promote [] to voter, rolling back: change replicas of r156 failed: storage/replica_raft.go:305: [n2,s2,r156/1:/{Table/90/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:43.034428 206960 storage/queue.go:1027 [n2,replicate,s2,r156/1:/{Table/90/2-Max}] change replicas of r156 failed: storage/replica_raft.go:305: [n2,s2,r156/1:/{Table/90/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:43.034901 208814 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72{-/1}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r114:/Table/72{-/1} [(n2,s2):2, next=3, gen=64] I190903 18:32:43.050635 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.054319 208814 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72{-/1}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:43.055404 208814 storage/store_snapshot.go:995 [n2,replicate,s2,r114/2:/Table/72{-/1}] sending LEARNER snapshot 6e0e79f7 at applied index 27 I190903 18:32:43.055578 208814 storage/store_snapshot.go:1038 [n2,replicate,s2,r114/2:/Table/72{-/1}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:43.061612 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.069332 208854 storage/replica_raftstorage.go:793 [n1,s1,r114/3:{-}] applying LEARNER snapshot [id=6e0e79f7 index=27] I190903 18:32:43.070734 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.080611 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.081822 208854 storage/replica_raftstorage.go:814 [n1,s1,r114/3:/Table/72{-/1}] applied LEARNER snapshot [total=12ms ingestion=4@12ms id=6e0e79f7 index=27] I190903 18:32:43.083124 208814 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72{-/1}] change replicas (add [(n1,s1):3] remove []): existing descriptor r114:/Table/72{-/1} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=65] I190903 18:32:43.123795 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.130805 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.131723 208814 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72{-/1}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 18:32:43.155519 208814 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r114:/Table/72{-/1} [(n2,s2):2, (n1,s1):3, next=4, gen=66] I190903 18:32:43.197184 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:43.198362 208814 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 18:32:43.198958 208814 storage/replica_raft.go:304 [n2,s2,r114/2:/Table/72{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:43.199839 208814 storage/replica_command.go:979 [n2,replicate,s2,r114/2:/Table/72{-/1}] could not promote [] to voter, rolling back: change replicas of r114 failed: storage/replica_raft.go:305: [n2,s2,r114/2:/Table/72{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 18:32:43.200610 208814 storage/queue.go:1027 [n2,replicate,s2,r114/2:/Table/72{-/1}] change replicas of r114 failed: storage/replica_raft.go:305: [n2,s2,r114/2:/Table/72{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 18:32:43.202816 209053 storage/replica_command.go:1430 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r157:/Table/90/1/{3-4} [(n2,s2):1, next=2, gen=134] I190903 18:32:43.208488 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.210069 209053 storage/replica_raft.go:291 [n2,s2,r157/1:/Table/90/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 18:32:43.212872 209053 storage/store_snapshot.go:995 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] sending LEARNER snapshot 1a66ea2c at applied index 18 I190903 18:32:43.213030 209053 storage/store_snapshot.go:1038 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:43.214642 209036 storage/replica_raftstorage.go:793 [n3,s3,r157/2:{-}] applying LEARNER snapshot [id=1a66ea2c index=18] I190903 18:32:43.237793 209036 storage/replica_raftstorage.go:814 [n3,s3,r157/2:/Table/90/1/{3-4}] applied LEARNER snapshot [total=23ms ingestion=4@23ms id=1a66ea2c index=18] I190903 18:32:43.243363 209053 storage/replica_command.go:1430 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r157:/Table/90/1/{3-4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=135] I190903 18:32:43.259727 209053 storage/replica_raft.go:291 [n2,s2,r157/1:/Table/90/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 18:32:43.260313 209053 storage/replica_command.go:1430 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r157:/Table/90/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=136] I190903 18:32:43.271445 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.275449 209053 storage/replica_raft.go:291 [n2,s2,r157/1:/Table/90/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 18:32:43.275484 209053 storage/replica_raft.go:304 [n2,s2,r157/1:/Table/90/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:43.277651 209053 storage/replica_command.go:979 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r157 failed: storage/replica_raft.go:305: [n2,s2,r157/1:/Table/90/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 18:32:43.277721 209053 storage/queue.go:1027 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] change replicas of r157 failed: storage/replica_raft.go:305: [n2,s2,r157/1:/Table/90/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 18:32:43.278171 209485 storage/replica_command.go:1430 [n2,replicate,s2,r147/1:/Table/88{-/1}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r147:/Table/88{-/1} [(n2,s2):1, next=2, gen=128] I190903 18:32:43.298749 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.302941 209485 storage/replica_raft.go:291 [n2,s2,r147/1:/Table/88{-/1}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:43.303895 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.303962 209485 storage/store_snapshot.go:995 [n2,replicate,s2,r147/1:/Table/88{-/1}] sending LEARNER snapshot 91ac1db3 at applied index 23 I190903 18:32:43.304131 209485 storage/store_snapshot.go:1038 [n2,replicate,s2,r147/1:/Table/88{-/1}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:43.305320 209546 storage/replica_raftstorage.go:793 [n1,s1,r147/2:{-}] applying LEARNER snapshot [id=91ac1db3 index=23] I190903 18:32:43.306836 209546 storage/replica_raftstorage.go:814 [n1,s1,r147/2:/Table/88{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=91ac1db3 index=23] I190903 18:32:43.307561 209485 storage/replica_command.go:1430 [n2,replicate,s2,r147/1:/Table/88{-/1}] change replicas (add [(n1,s1):2] remove []): existing descriptor r147:/Table/88{-/1} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=129] I190903 18:32:43.315463 209485 storage/replica_raft.go:291 [n2,s2,r147/1:/Table/88{-/1}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:43.316124 209485 storage/replica_command.go:1430 [n2,replicate,s2,r147/1:/Table/88{-/1}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r147:/Table/88{-/1} [(n2,s2):1, (n1,s1):2, next=3, gen=130] I190903 18:32:43.319811 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.323098 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.326870 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.329907 209485 storage/replica_raft.go:291 [n2,s2,r147/1:/Table/88{-/1}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:43.331594 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 18:32:43.331598 209485 storage/replica_raft.go:304 [n2,s2,r147/1:/Table/88{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:43.333195 209485 storage/replica_command.go:979 [n2,replicate,s2,r147/1:/Table/88{-/1}] could not promote [] to voter, rolling back: change replicas of r147 failed: storage/replica_raft.go:305: [n2,s2,r147/1:/Table/88{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:43.333289 209485 storage/queue.go:1027 [n2,replicate,s2,r147/1:/Table/88{-/1}] change replicas of r147 failed: storage/replica_raft.go:305: [n2,s2,r147/1:/Table/88{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:43.334199 209723 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r158:/Table/90/{1/4-2} [(n2,s2):1, next=2, gen=134] I190903 18:32:43.336678 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.340977 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.342216 209723 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:43.344688 209723 storage/store_snapshot.go:995 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] sending LEARNER snapshot b66e337f at applied index 15 I190903 18:32:43.344897 209723 storage/store_snapshot.go:1038 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:43.345423 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.349168 209801 storage/replica_raftstorage.go:793 [n1,s1,r158/2:{-}] applying LEARNER snapshot [id=b66e337f index=15] I190903 18:32:43.350920 209801 storage/replica_raftstorage.go:814 [n1,s1,r158/2:/Table/90/{1/4-2}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=b66e337f index=15] I190903 18:32:43.351035 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.351756 209723 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r158:/Table/90/{1/4-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=135] I190903 18:32:43.363926 209723 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:43.364704 209723 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r158:/Table/90/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=136] I190903 18:32:43.368300 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.381016 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.397733 209723 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:43.397774 209723 storage/replica_raft.go:304 [n2,s2,r158/1:/Table/90/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:43.399633 209723 storage/replica_command.go:979 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r158 failed: storage/replica_raft.go:305: [n2,s2,r158/1:/Table/90/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:43.399693 209723 storage/queue.go:1027 [n2,replicate,s2,r158/1:/Table/90/{1/4-2}] change replicas of r158 failed: storage/replica_raft.go:305: [n2,s2,r158/1:/Table/90/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:43.400900 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.407390 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 18:32:43.411462 210035 storage/queue.go:1027 [n3,replicate,s3,r157/2:/Table/90/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:26] E190903 18:32:43.467869 210230 storage/queue.go:1027 [n3,replicate,s3,r156/2:/{Table/90/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:43.469902 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.477434 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.482806 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 E190903 18:32:43.489879 210328 storage/queue.go:1027 [n1,replicate,s1,r158/2:/Table/90/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:43.490349 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 18:32:43.502590 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 18:32:43.506483 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 18:32:43.528235 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=90 (single col list partitioning) version=2 mtime=2019-09-03 18:32:43.512630021 +0000 UTC I190903 18:32:43.540536 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:43.554160 210491 sql/lease.go:923 new lease: 90("single col list partitioning") ver=2:1567535863.620702193,0, refcount=0 I190903 18:32:43.570715 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 90 1}] I190903 18:32:43.635723 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=90 (single col list partitioning) version=3 mtime=2019-09-03 18:32:43.634903181 +0000 UTC I190903 18:32:43.647525 211368 sql/lease.go:923 new lease: 90("single col list partitioning") ver=3:1567535865.864017496,0, refcount=0 I190903 18:32:43.653898 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 90 2}] I190903 18:32:43.675367 211632 storage/replica_command.go:1430 [n3,replicate,s3,r156/2:/{Table/90/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r156:/{Table/90/2-Max} [(n2,s2):1, (n3,s3):2, next=3, gen=134] I190903 18:32:43.686953 211632 storage/replica_raft.go:291 [n3,s3,r156/2:/{Table/90/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 18:32:43.688858 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=90 (single col list partitioning) version=4 mtime=2019-09-03 18:32:43.685086066 +0000 UTC I190903 18:32:43.696961 211595 storage/store.go:2593 [n2,replicaGC,s2,r156/1:/{Table/90/2-Max}] removing replica r156/1 I190903 18:32:43.697573 211595 storage/replica_destroy.go:152 [n2,replicaGC,s2,r156/1:/{Table/90/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:43.705486 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 90, info: {TableName:data.public."single col list partitioning" Statement:ALTER TABLE "single col list partitioning" PARTITION BY LIST (a) (PARTITION p3 VALUES IN ((3)), PARTITION p4 VALUES IN ((4))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:43.715189 211848 sql/lease.go:923 new lease: 90("single col list partitioning") ver=4:1567535865.865017496,0, refcount=0 I190903 18:32:43.715681 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 90 3}] I190903 18:32:43.781195 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 90, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:43.784534 212854 storage/replica_command.go:284 [n1,split,s1,r158/2:/Table/90/{1/4-2}] initiating a split of this range at key /Table/90/1/5 [r98] (zone config) E190903 18:32:43.834096 212945 storage/queue.go:1027 [n1,replicate,s1,r114/3:/Table/72{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:34] E190903 18:32:43.834303 213062 storage/queue.go:1027 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] no removable replicas from range that needs a removal: [1*:28, 2:0] E190903 18:32:43.847915 213158 storage/queue.go:1027 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] no removable replicas from range that needs a removal: [1*:29, 2:0] I190903 18:32:43.855698 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 90, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:43.882305 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 90, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:43.882416 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.887152 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.892489 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.897846 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.900989 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.904349 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.909624 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:43.916348 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.922311 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:43.928500 214206 storage/queue.go:1027 [n1,replicate,s1,r147/2:/Table/88{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 18:32:43.934960 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.940043 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:43.956498 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:43.980804 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:43.991615 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:44.003877 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:44.010766 214237 storage/replica_command.go:1430 [n1,replicate,s1,r114/3:/Table/72{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r114:/Table/72{-/1} [(n2,s2):2, (n1,s1):3, next=4, gen=66] I190903 18:32:44.011028 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.017864 214429 storage/replica_command.go:1430 [n2,replicate,s2,r157/1:/Table/90/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r157:/Table/90/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=136] I190903 18:32:44.024185 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.045925 214429 storage/replica_raft.go:291 [n2,s2,r157/1:/Table/90/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):1] next=3 I190903 18:32:44.045926 214237 storage/replica_raft.go:291 [n1,s1,r114/3:/Table/72{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:44.046358 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:44.051820 214558 storage/replica_command.go:1430 [n1,replicate,s1,r98/2:/Table/90/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r98:/Table/90/{1/5-2} [(n2,s2):1, (n1,s1):2, next=3, gen=137] I190903 18:32:44.052744 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.056487 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.059864 214577 storage/store.go:2593 [n2,replicaGC,s2,r114/2:/Table/72{-/1}] removing replica r114/2 I190903 18:32:44.059915 214629 storage/store.go:2593 [n3,replicaGC,s3,r157/2:/Table/90/1/{3-4}] removing replica r157/2 I190903 18:32:44.060355 214577 storage/replica_destroy.go:152 [n2,replicaGC,s2,r114/2:/Table/72{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:44.061797 214629 storage/replica_destroy.go:152 [n3,replicaGC,s3,r157/2:/Table/90/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:44.067553 214558 storage/replica_raft.go:291 [n1,s1,r98/2:/Table/90/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:44.067805 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.071703 214740 storage/replica_command.go:1430 [n1,replicate,s1,r147/2:/Table/88{-/1}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r147:/Table/88{-/1} [(n2,s2):1, (n1,s1):2, next=3, gen=130] I190903 18:32:44.077410 214690 storage/store.go:2593 [n2,replicaGC,s2,r98/1:/Table/90/{1/5-2}] removing replica r98/1 I190903 18:32:44.077701 214690 storage/replica_destroy.go:152 [n2,replicaGC,s2,r98/1:/Table/90/{1/5-2}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:44.090320 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:44.096408 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:44.098813 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.109692 214740 storage/replica_raft.go:291 [n1,s1,r147/2:/Table/88{-/1}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:44.114759 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.146861 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.163344 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:44.166163 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.172902 214888 storage/store.go:2593 [n2,replicaGC,s2,r147/1:/Table/88{-/1}] removing replica r147/1 I190903 18:32:44.173526 214888 storage/replica_destroy.go:152 [n2,replicaGC,s2,r147/1:/Table/88{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:44.184994 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 E190903 18:32:44.192996 215125 storage/queue.go:1027 [n1,replicate,s1,r158/2:/Table/90/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:31] I190903 18:32:44.217976 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.241287 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.280041 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:44.285578 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:44.360118 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:44.362299 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:44.432826 215758 storage/replica_command.go:1430 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r155:/Table/90/1{-/3} [(n2,s2):1, next=2, gen=133] I190903 18:32:44.439701 215758 storage/replica_raft.go:291 [n2,s2,r155/1:/Table/90/1{-/3}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 18:32:44.441008 215758 storage/store_snapshot.go:995 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] sending LEARNER snapshot ebdbf08d at applied index 26 I190903 18:32:44.442102 215758 storage/store_snapshot.go:1038 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:44.443717 215702 storage/replica_raftstorage.go:793 [n1,s1,r155/2:{-}] applying LEARNER snapshot [id=ebdbf08d index=26] I190903 18:32:44.444814 215702 storage/replica_raftstorage.go:814 [n1,s1,r155/2:/Table/90/1{-/3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ebdbf08d index=26] I190903 18:32:44.445347 215758 storage/replica_command.go:1430 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] change replicas (add [(n1,s1):2] remove []): existing descriptor r155:/Table/90/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=134] I190903 18:32:44.453917 215758 storage/replica_raft.go:291 [n2,s2,r155/1:/Table/90/1{-/3}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 18:32:44.454426 215758 storage/replica_command.go:1430 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r155:/Table/90/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=135] I190903 18:32:44.468239 215758 storage/replica_raft.go:291 [n2,s2,r155/1:/Table/90/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 18:32:44.468325 215758 storage/replica_raft.go:304 [n2,s2,r155/1:/Table/90/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:44.469589 215758 storage/replica_command.go:979 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] could not promote [] to voter, rolling back: change replicas of r155 failed: storage/replica_raft.go:305: [n2,s2,r155/1:/Table/90/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:44.469650 215758 storage/queue.go:1027 [n2,replicate,s2,r155/1:/Table/90/1{-/3}] change replicas of r155 failed: storage/replica_raft.go:305: [n2,s2,r155/1:/Table/90/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:44.470894 215914 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r158:/Table/90/1/{4-5} [(n2,s2):1, (n1,s1):2, next=3, gen=137] I190903 18:32:44.476507 215914 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):1] next=3 I190903 18:32:44.477524 215914 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r158:/Table/90/1/{4-5} [(n2,s2):1, next=3, gen=138] I190903 18:32:44.485494 215862 storage/store.go:2593 [n1,replicaGC,s1,r158/2:/Table/90/1/{4-5}] removing replica r158/2 I190903 18:32:44.486104 215862 storage/replica_destroy.go:152 [n1,replicaGC,s1,r158/2:/Table/90/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:44.490645 215914 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):1 (n3,s3):3LEARNER] next=4 I190903 18:32:44.492084 215914 storage/store_snapshot.go:995 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] sending LEARNER snapshot bd4aa439 at applied index 39 I190903 18:32:44.492817 215914 storage/store_snapshot.go:1038 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:44.494071 214732 storage/replica_raftstorage.go:793 [n3,s3,r158/3:{-}] applying LEARNER snapshot [id=bd4aa439 index=39] I190903 18:32:44.495524 214732 storage/replica_raftstorage.go:814 [n3,s3,r158/3:/Table/90/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=bd4aa439 index=39] I190903 18:32:44.496216 215914 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] change replicas (add [(n3,s3):3] remove []): existing descriptor r158:/Table/90/1/{4-5} [(n2,s2):1, (n3,s3):3LEARNER, next=4, gen=139] I190903 18:32:44.501205 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:44.503420 215914 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):1 (n3,s3):3] next=4 I190903 18:32:44.504078 215914 storage/replica_command.go:1430 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r158:/Table/90/1/{4-5} [(n2,s2):1, (n3,s3):3, next=4, gen=140] I190903 18:32:44.518001 215914 storage/replica_raft.go:291 [n2,s2,r158/1:/Table/90/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 E190903 18:32:44.518058 215914 storage/replica_raft.go:304 [n2,s2,r158/1:/Table/90/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 18:32:44.520574 215914 storage/replica_command.go:979 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r158 failed: storage/replica_raft.go:305: [n2,s2,r158/1:/Table/90/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:44.520619 215914 storage/queue.go:1027 [n2,replicate,s2,r158/1:/Table/90/1/{4-5}] change replicas of r158 failed: storage/replica_raft.go:305: [n2,s2,r158/1:/Table/90/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 18:32:44.702642 216313 storage/queue.go:1027 [n3,replicate,s3,r158/3:/Table/90/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 3*:47] I190903 18:32:44.772622 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:44.776689 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:44.778449 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:44.886676 216879 storage/queue.go:1027 [n1,replicate,s1,r155/2:/Table/90/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:33] I190903 18:32:45.040777 217529 storage/replica_command.go:1430 [n3,replicate,s3,r158/3:/Table/90/1/{4-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r158:/Table/90/1/{4-5} [(n2,s2):1, (n3,s3):3, next=4, gen=140] I190903 18:32:45.046179 217529 storage/replica_raft.go:291 [n3,s3,r158/3:/Table/90/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):3] next=4 I190903 18:32:45.054487 217439 storage/store.go:2593 [n2,replicaGC,s2,r158/1:/Table/90/1/{4-5}] removing replica r158/1 I190903 18:32:45.054980 217439 storage/replica_destroy.go:152 [n2,replicaGC,s2,r158/1:/Table/90/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:45.254541 217585 storage/replica_command.go:1430 [n1,replicate,s1,r155/2:/Table/90/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r155:/Table/90/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=135] I190903 18:32:45.263337 217585 storage/replica_raft.go:291 [n1,s1,r155/2:/Table/90/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 18:32:45.316970 218063 storage/store.go:2593 [n2,replicaGC,s2,r155/1:/Table/90/1{-/3}] removing replica r155/1 I190903 18:32:45.317491 218063 storage/replica_destroy.go:152 [n2,replicaGC,s2,r155/1:/Table/90/1{-/3}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:45.318378 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.321222 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:45.326613 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:45.329849 204588 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 === RUN TestRepartitioning/single_col_list_partitioning/unpartitioned I190903 18:32:45.367029 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=90 (single col list partitioning) version=5 mtime=2019-09-03 18:32:45.331858943 +0000 UTC I190903 18:32:45.391848 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 89, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning"]} I190903 18:32:45.400290 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 90 4}] I190903 18:32:45.462010 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=90 (single col list partitioning) version=6 mtime=2019-09-03 18:32:45.46130465 +0000 UTC I190903 18:32:45.509395 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 91, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:45.525736 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 92, info: {TableName:data.public."single col list partitioning" Statement:CREATE TABLE "single col list partitioning" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p3 VALUES IN (3), PARTITION p4 VALUES IN (4)) User:root} I190903 18:32:45.527729 220404 storage/replica_command.go:284 [n3,split,s3,r156/2:/{Table/90/2-Max}] initiating a split of this range at key /Table/92 [r179] (zone config) I190903 18:32:45.528406 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:45.554190 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 92, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:45.555789 220794 storage/replica_command.go:284 [n3,split,s3,r179/1:/{Table/92-Max}] initiating a split of this range at key /Table/92/1 [r180] (zone config) I190903 18:32:45.565041 221109 storage/replica_command.go:284 [n3,split,s3,r180/1:/{Table/92/1-Max}] initiating a split of this range at key /Table/92/2 [r181] (zone config) I190903 18:32:45.569651 174 storage/queue.go:518 [n1,s1,r8/1:/Table/1{2-3}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:45.573063 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 92, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:45.582538 221504 storage/replica_command.go:284 [n3,split,s3,r180/1:/Table/92/{1-2}] initiating a split of this range at key /Table/92/1/3 [r182] (zone config) I190903 18:32:45.594656 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 92, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:45.594768 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:45.596160 221630 storage/replica_command.go:284 [n3,split,s3,r182/1:/Table/92/{1/3-2}] initiating a split of this range at key /Table/92/1/4 [r183] (zone config) I190903 18:32:45.596757 221896 storage/replica_command.go:1430 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r180:/Table/92/1{-/3} [(n3,s3):1, next=2, gen=139] I190903 18:32:45.623964 221896 storage/replica_raft.go:291 [n3,s3,r180/1:/Table/92/1{-/3}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:45.625143 221896 storage/store_snapshot.go:995 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] sending LEARNER snapshot 7e4296fc at applied index 20 I190903 18:32:45.625330 221600 storage/replica_command.go:284 [n3,split,s3,r183/1:/Table/92/{1/4-2}] initiating a split of this range at key /Table/92/1/5 [r184] (zone config) I190903 18:32:45.625331 221896 storage/store_snapshot.go:1038 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:45.627720 222075 storage/replica_raftstorage.go:793 [n1,s1,r180/2:{-}] applying LEARNER snapshot [id=7e4296fc index=20] I190903 18:32:45.633119 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:45.641823 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:45.642968 222075 storage/replica_raftstorage.go:814 [n1,s1,r180/2:/Table/92/1{-/3}] applied LEARNER snapshot [total=15ms ingestion=4@15ms id=7e4296fc index=20] I190903 18:32:45.643556 221896 storage/replica_command.go:1430 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] change replicas (add [(n1,s1):2] remove []): existing descriptor r180:/Table/92/1{-/3} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=140] I190903 18:32:45.647109 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.649029 221896 storage/replica_raft.go:291 [n3,s3,r180/1:/Table/92/1{-/3}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:45.649640 221896 storage/replica_command.go:1430 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r180:/Table/92/1{-/3} [(n3,s3):1, (n1,s1):2, next=3, gen=141] I190903 18:32:45.653233 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:45.658087 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.661769 221896 storage/replica_raft.go:291 [n3,s3,r180/1:/Table/92/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:45.661824 221896 storage/replica_raft.go:304 [n3,s3,r180/1:/Table/92/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:45.662071 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.663177 221896 storage/replica_command.go:979 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] could not promote [] to voter, rolling back: change replicas of r180 failed: storage/replica_raft.go:305: [n3,s3,r180/1:/Table/92/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:45.663220 221896 storage/queue.go:1027 [n3,replicate,s3,r180/1:/Table/92/1{-/3}] change replicas of r180 failed: storage/replica_raft.go:305: [n3,s3,r180/1:/Table/92/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:45.663652 222285 storage/replica_command.go:1430 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r182:/Table/92/1/{3-4} [(n3,s3):1, next=2, gen=140] I190903 18:32:45.665663 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.669012 222285 storage/replica_raft.go:291 [n3,s3,r182/1:/Table/92/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:45.669460 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.670121 222285 storage/store_snapshot.go:995 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] sending LEARNER snapshot fb14ffec at applied index 17 I190903 18:32:45.670277 222285 storage/store_snapshot.go:1038 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:45.671568 222094 storage/replica_raftstorage.go:793 [n2,s2,r182/2:{-}] applying LEARNER snapshot [id=fb14ffec index=17] I190903 18:32:45.672797 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.673363 222094 storage/replica_raftstorage.go:814 [n2,s2,r182/2:/Table/92/1/{3-4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=fb14ffec index=17] I190903 18:32:45.673799 222285 storage/replica_command.go:1430 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r182:/Table/92/1/{3-4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=141] I190903 18:32:45.677153 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.682785 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.684372 222285 storage/replica_raft.go:291 [n3,s3,r182/1:/Table/92/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:45.684963 222285 storage/replica_command.go:1430 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r182:/Table/92/1/{3-4} [(n3,s3):1, (n2,s2):2, next=3, gen=142] I190903 18:32:45.688172 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.697908 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.700785 222285 storage/replica_raft.go:291 [n3,s3,r182/1:/Table/92/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:45.700822 222285 storage/replica_raft.go:304 [n3,s3,r182/1:/Table/92/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:45.701629 222285 storage/replica_command.go:979 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r182 failed: storage/replica_raft.go:305: [n3,s3,r182/1:/Table/92/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:45.701661 222285 storage/queue.go:1027 [n3,replicate,s3,r182/1:/Table/92/1/{3-4}] change replicas of r182 failed: storage/replica_raft.go:305: [n3,s3,r182/1:/Table/92/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:45.702071 222606 storage/replica_command.go:1430 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r184:/Table/92/{1/5-2} [(n3,s3):1, next=2, gen=141] I190903 18:32:45.702249 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.706663 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:45.714765 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.716126 222606 storage/replica_raft.go:291 [n3,s3,r184/1:/Table/92/{1/5-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:45.717421 222606 storage/store_snapshot.go:995 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] sending LEARNER snapshot 64ad53fc at applied index 15 I190903 18:32:45.717693 222606 storage/store_snapshot.go:1038 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:45.718171 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.718480 222618 storage/replica_raftstorage.go:793 [n1,s1,r184/2:{-}] applying LEARNER snapshot [id=64ad53fc index=15] I190903 18:32:45.719925 222618 storage/replica_raftstorage.go:814 [n1,s1,r184/2:/Table/92/{1/5-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=64ad53fc index=15] I190903 18:32:45.721681 222606 storage/replica_command.go:1430 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r184:/Table/92/{1/5-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=142] I190903 18:32:45.722413 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.727193 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.729507 222606 storage/replica_raft.go:291 [n3,s3,r184/1:/Table/92/{1/5-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:45.730132 222606 storage/replica_command.go:1430 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r184:/Table/92/{1/5-2} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:45.731577 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.738183 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.745343 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.745482 222606 storage/replica_raft.go:291 [n3,s3,r184/1:/Table/92/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:45.745546 222606 storage/replica_raft.go:304 [n3,s3,r184/1:/Table/92/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:45.746574 222606 storage/replica_command.go:979 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r184 failed: storage/replica_raft.go:305: [n3,s3,r184/1:/Table/92/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:45.746664 222606 storage/queue.go:1027 [n3,replicate,s3,r184/1:/Table/92/{1/5-2}] change replicas of r184 failed: storage/replica_raft.go:305: [n3,s3,r184/1:/Table/92/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:45.750936 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.759628 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:45.772399 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:45.775816 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:45.795405 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:45.832358 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:45.903516 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:46.043889 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:46.069784 223802 storage/queue.go:1027 [n1,replicate,s1,r184/2:/Table/92/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:46.316041 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 18:32:46.369139 224608 storage/queue.go:1027 [n1,replicate,s1,r180/2:/Table/92/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:27] E190903 18:32:46.522153 224976 storage/queue.go:1027 [n2,replicate,s2,r182/2:/Table/92/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 18:32:46.760373 225695 storage/replica_command.go:1430 [n1,replicate,s1,r184/2:/Table/92/{1/5-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r184:/Table/92/{1/5-2} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:46.777081 225695 storage/replica_raft.go:291 [n1,s1,r184/2:/Table/92/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:46.806849 281 server/status/runtime.go:498 [n1] runtime stats: 268 MiB RSS, 952 goroutines, 84 MiB/17 MiB/139 MiB GO alloc/idle/total, 52 MiB/108 MiB CGO alloc/total, 7850.7 CGO/sec, 82.6/10.3 %(u/s)time, 0.4 %gc (11x), 89 MiB/89 MiB (r/w)net I190903 18:32:46.842359 225709 storage/store.go:2593 [n3,replicaGC,s3,r184/1:/Table/92/{1/5-2}] removing replica r184/1 I190903 18:32:46.842764 225709 storage/replica_destroy.go:152 [n3,replicaGC,s3,r184/1:/Table/92/{1/5-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:46.856601 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:46.929400 226158 storage/replica_command.go:1430 [n1,replicate,s1,r180/2:/Table/92/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r180:/Table/92/1{-/3} [(n3,s3):1, (n1,s1):2, next=3, gen=141] I190903 18:32:46.937375 226158 storage/replica_raft.go:291 [n1,s1,r180/2:/Table/92/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:46.947623 226196 storage/store.go:2593 [n3,replicaGC,s3,r180/1:/Table/92/1{-/3}] removing replica r180/1 I190903 18:32:46.947977 226196 storage/replica_destroy.go:152 [n3,replicaGC,s3,r180/1:/Table/92/1{-/3}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:47.031638 226469 storage/replica_command.go:1430 [n2,replicate,s2,r182/2:/Table/92/1/{3-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r182:/Table/92/1/{3-4} [(n3,s3):1, (n2,s2):2, next=3, gen=142] I190903 18:32:47.038954 226469 storage/replica_raft.go:291 [n2,s2,r182/2:/Table/92/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:47.047517 226418 storage/store.go:2593 [n3,replicaGC,s3,r182/1:/Table/92/1/{3-4}] removing replica r182/1 I190903 18:32:47.047934 226418 storage/replica_destroy.go:152 [n3,replicaGC,s3,r182/1:/Table/92/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:47.229056 1651 server/status/runtime.go:498 [n2] runtime stats: 270 MiB RSS, 961 goroutines, 51 MiB/34 MiB/139 MiB GO alloc/idle/total, 53 MiB/110 MiB CGO alloc/total, 7564.9 CGO/sec, 79.0/9.8 %(u/s)time, 0.3 %gc (11x), 88 MiB/88 MiB (r/w)net I190903 18:32:47.497040 2361 server/status/runtime.go:498 [n3] runtime stats: 271 MiB RSS, 957 goroutines, 61 MiB/26 MiB/139 MiB GO alloc/idle/total, 53 MiB/110 MiB CGO alloc/total, 6972.6 CGO/sec, 72.3/9.5 %(u/s)time, 0.3 %gc (10x), 88 MiB/88 MiB (r/w)net I190903 18:32:47.860721 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 18:32:47.862950 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 18:32:47.866438 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 18:32:47.874684 218372 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 18:32:47.882195 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=92 (unpartitioned) version=2 mtime=2019-09-03 18:32:47.876748826 +0000 UTC I190903 18:32:47.896356 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:47.916468 228622 sql/lease.go:923 new lease: 92("unpartitioned") ver=2:1567535875.343097400,0, refcount=0 I190903 18:32:47.916715 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 92 1}] I190903 18:32:47.972957 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=92 (unpartitioned) version=3 mtime=2019-09-03 18:32:47.972408583 +0000 UTC I190903 18:32:47.980258 2377 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 18:32:47.988378 229221 sql/lease.go:923 new lease: 92("unpartitioned") ver=3:1567535875.344097400,0, refcount=0 I190903 18:32:47.988432 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 92 2}] I190903 18:32:48.036903 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=92 (unpartitioned) version=4 mtime=2019-09-03 18:32:48.0253831 +0000 UTC I190903 18:32:48.061582 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 92, info: {TableName:data.public.unpartitioned Statement:ALTER TABLE unpartitioned PARTITION BY NOTHING User:root MutationID:0 CascadeDroppedViews:[]} E190903 18:32:48.063696 230170 storage/queue.go:1027 [n3,merge,s3,r183/1:/Table/92/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 18:32:48.066159 230327 storage/replica_command.go:1430 [n2,s2,r182/2:/Table/92/1/{3-4}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r182:/Table/92/1/{3-4} [(n2,s2):2, next=3, gen=143] E190903 18:32:48.069119 230495 storage/queue.go:1027 [n2,merge,s2,r182/2:/Table/92/1/{3-4}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 18:32:48.073020 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 92 3}] I190903 18:32:48.073421 229723 sql/lease.go:923 new lease: 92("unpartitioned") ver=4:1567535875.345097400,0, refcount=0 I190903 18:32:48.089579 230327 storage/replica_raft.go:291 [n2,s2,r182/2:/Table/92/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 18:32:48.092030 230327 storage/store_snapshot.go:995 [n2,s2,r182/2:/Table/92/1/{3-4}] sending LEARNER snapshot 836bf9e4 at applied index 36 I190903 18:32:48.092199 230327 storage/store_snapshot.go:1038 [n2,s2,r182/2:/Table/92/1/{3-4}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:48.093348 230599 storage/replica_raftstorage.go:793 [n1,s1,r182/3:{-}] applying LEARNER snapshot [id=836bf9e4 index=36] I190903 18:32:48.103021 230599 storage/replica_raftstorage.go:814 [n1,s1,r182/3:/Table/92/1/{3-4}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=836bf9e4 index=36] I190903 18:32:48.108507 230327 storage/replica_command.go:1430 [n2,s2,r182/2:/Table/92/1/{3-4}] change replicas (add [(n1,s1):3] remove []): existing descriptor r182:/Table/92/1/{3-4} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=144] === RUN TestRepartitioning/single_col_list_partitioning_-_DEFAULT/unpartitioned I190903 18:32:48.143760 230327 storage/replica_raft.go:291 [n2,s2,r182/2:/Table/92/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 E190903 18:32:48.170543 230332 storage/queue.go:1027 [n1,replicate,s1,r182/3:/Table/92/1/{3-4}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 18:32:48.227472 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=92 (unpartitioned) version=5 mtime=2019-09-03 18:32:48.119600611 +0000 UTC E190903 18:32:48.238584 231085 storage/queue.go:1027 [n3,merge,s3,r183/1:/Table/92/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 18:32:48.239093 231086 storage/replica_command.go:1430 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r183:/Table/92/1/{4-5} [(n3,s3):1, next=2, gen=141] I190903 18:32:48.242295 1614 storage/queue.go:518 [n2,s2,r9/2:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 18:32:48.245167 231086 storage/replica_raft.go:291 [n3,s3,r183/1:/Table/92/1/{4-5}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:48.246746 231086 storage/store_snapshot.go:995 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] sending LEARNER snapshot 39a82254 at applied index 23 I190903 18:32:48.246944 231086 storage/store_snapshot.go:1038 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:48.248260 231009 storage/replica_raftstorage.go:793 [n1,s1,r183/2:{-}] applying LEARNER snapshot [id=39a82254 index=23] I190903 18:32:48.249853 231009 storage/replica_raftstorage.go:814 [n1,s1,r183/2:/Table/92/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=39a82254 index=23] I190903 18:32:48.250390 231086 storage/replica_command.go:1430 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] change replicas (add [(n1,s1):2] remove []): existing descriptor r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=142] I190903 18:32:48.251016 230923 storage/replica_command.go:1430 [n1,s1,r182/3:/Table/92/1/{3-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r182:/Table/92/1/{3-4} [(n2,s2):2, (n1,s1):3, next=4, gen=145] I190903 18:32:48.253407 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 91, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public.unpartitioned]} I190903 18:32:48.261555 1673 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:48.264432 231086 storage/replica_raft.go:291 [n3,s3,r183/1:/Table/92/1/{4-5}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:48.264872 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 92 4}] I190903 18:32:48.264915 230923 storage/replica_raft.go:291 [n1,s1,r182/3:/Table/92/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 18:32:48.267635 231086 storage/replica_command.go:1430 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:48.271095 230326 storage/replica_command.go:609 [n1,merge,s1,r180/2:/Table/92/1{-/3}] initiating a merge of r182:/Table/92/1/{3-4} [(n1,s1):3, next=4, gen=146] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:48.273108 231151 storage/replica_command.go:979 [n3,s3,r183/1:/Table/92/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r183 failed: descriptor changed: [expected] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=142] != [actual] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] E190903 18:32:48.273189 231151 storage/queue.go:1027 [n3,merge,s3,r183/1:/Table/92/1/{4-5}] removing learners from r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=142]: change replicas of r183 failed: descriptor changed: [expected] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=142] != [actual] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:48.279594 230909 storage/store.go:2593 [n2,replicaGC,s2,r182/2:/Table/92/1/{3-4}] removing replica r182/2 I190903 18:32:48.280422 230909 storage/replica_destroy.go:152 [n2,replicaGC,s2,r182/2:/Table/92/1/{3-4}] removed 7 (0+7) keys in 1ms [clear=1ms commit=0ms] I190903 18:32:48.282285 231086 storage/replica_raft.go:291 [n3,s3,r183/1:/Table/92/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:48.282331 231086 storage/replica_raft.go:304 [n3,s3,r183/1:/Table/92/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:48.284022 231086 storage/replica_command.go:979 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r183 failed: storage/replica_raft.go:305: [n3,s3,r183/1:/Table/92/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:48.284066 231086 storage/queue.go:1027 [n3,replicate,s3,r183/1:/Table/92/1/{4-5}] change replicas of r183 failed: storage/replica_raft.go:305: [n3,s3,r183/1:/Table/92/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:48.284796 186 storage/store.go:2593 [n1,s1,r180/2:/Table/92/1{-/3}] removing replica r182/3 I190903 18:32:48.327653 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=92 (unpartitioned) version=6 mtime=2019-09-03 18:32:48.326830378 +0000 UTC I190903 18:32:48.343961 231629 storage/replica_command.go:1430 [n1,replicate,s1,r183/2:/Table/92/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:48.359388 231629 storage/replica_raft.go:291 [n1,s1,r183/2:/Table/92/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:48.366491 231895 storage/replica_command.go:979 [n1,s1,r183/2:/Table/92/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r183 failed: descriptor changed: [expected] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] != [actual] r183:/Table/92/1/{4-5} [(n1,s1):2, next=3, gen=144] W190903 18:32:48.367075 231672 storage/replica_command.go:2033 [n3,s3,r183/1:/Table/92/1/{4-5}] while removing target n3,s3: change replicas of r183 failed: descriptor changed: [expected] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] != [actual] r183:/Table/92/1/{4-5} [(n1,s1):2, next=3, gen=144] E190903 18:32:48.367231 231671 storage/queue.go:1027 [n1,merge,s1,r180/2:/Table/92/1{-/4}] change replicas of r183 failed: descriptor changed: [expected] r183:/Table/92/1/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=143] != [actual] r183:/Table/92/1/{4-5} [(n1,s1):2, next=3, gen=144] I190903 18:32:48.399292 232112 storage/replica_command.go:609 [n1,merge,s1,r183/2:/Table/92/1/{4-5}] initiating a merge of r184:/Table/92/{1/5-2} [(n1,s1):2, next=3, gen=144] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:48.409665 250 storage/store.go:2593 [n1,s1,r183/2:/Table/92/1/{4-5}] removing replica r184/2 I190903 18:32:48.411252 232538 storage/replica_command.go:609 [n1,merge,s1,r180/2:/Table/92/1{-/4}] initiating a merge of r183:/Table/92/{1/4-2} [(n1,s1):2, next=3, gen=145] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:48.428447 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 93, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:48.429341 248 storage/store.go:2593 [n1,s1,r180/2:/Table/92/1{-/4}] removing replica r183/2 I190903 18:32:48.432254 231990 storage/store.go:2593 [n3,replicaGC,s3,r183/1:/Table/92/1/{4-5}] removing replica r183/1 I190903 18:32:48.432653 231990 storage/replica_destroy.go:152 [n3,replicaGC,s3,r183/1:/Table/92/1/{4-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:48.464198 233439 storage/replica_command.go:284 [n3,split,s3,r181/1:/{Table/92/2-Max}] initiating a split of this range at key /Table/94 [r185] (zone config) I190903 18:32:48.464405 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 94, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:CREATE TABLE "single col list partitioning - DEFAULT" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p4 VALUES IN (4), PARTITION p5 VALUES IN (5), PARTITION pd VALUES IN (DEFAULT)) User:root} I190903 18:32:48.509419 233779 storage/replica_command.go:284 [n3,split,s3,r185/1:/{Table/94-Max}] initiating a split of this range at key /Table/94/1/4 [r186] (zone config) I190903 18:32:48.523258 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 94, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 18:32:48.539612 233874 storage/replica_command.go:284 [n3,split,s3,r186/1:/{Table/94/1/4-Max}] initiating a split of this range at key /Table/94/1/5 [r187] (zone config) I190903 18:32:48.557744 234204 storage/replica_command.go:1430 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r186:/Table/94/1/{4-5} [(n3,s3):1, next=2, gen=141] I190903 18:32:48.561968 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 94, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 18:32:48.566058 234001 storage/replica_command.go:284 [n3,split,s3,r187/1:/{Table/94/1/5-Max}] initiating a split of this range at key /Table/94/1/6 [r188] (zone config) I190903 18:32:48.589794 234204 storage/replica_raft.go:291 [n3,s3,r186/1:/Table/94/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:48.591347 234204 storage/store_snapshot.go:995 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] sending LEARNER snapshot a2b269f7 at applied index 17 I190903 18:32:48.591631 234204 storage/store_snapshot.go:1038 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:48.593230 234705 storage/replica_raftstorage.go:793 [n2,s2,r186/2:{-}] applying LEARNER snapshot [id=a2b269f7 index=17] I190903 18:32:48.595345 234705 storage/replica_raftstorage.go:814 [n2,s2,r186/2:/Table/94/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=a2b269f7 index=17] I190903 18:32:48.598663 234204 storage/replica_command.go:1430 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r186:/Table/94/1/{4-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=142] I190903 18:32:48.611652 234163 storage/replica_command.go:284 [n3,split,s3,r185/1:/Table/94{-/1/4}] initiating a split of this range at key /Table/94/1 [r189] (zone config) I190903 18:32:48.612474 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "set_zone_config", target: 94, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 18:32:48.612796 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.627135 141 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 18:32:48.635741 235125 storage/replica_command.go:284 [n3,split,s3,r188/1:/{Table/94/1/6-Max}] initiating a split of this range at key /Table/94/2 [r190] (zone config) I190903 18:32:48.669395 234204 storage/replica_raft.go:291 [n3,s3,r186/1:/Table/94/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 18:32:48.670070 234204 storage/replica_command.go:1430 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r186:/Table/94/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=143] I190903 18:32:48.684034 234204 storage/replica_raft.go:291 [n3,s3,r186/1:/Table/94/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 18:32:48.684082 234204 storage/replica_raft.go:304 [n3,s3,r186/1:/Table/94/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:48.685475 234204 storage/replica_command.go:979 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r186 failed: storage/replica_raft.go:305: [n3,s3,r186/1:/Table/94/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 18:32:48.685513 234204 storage/queue.go:1027 [n3,replicate,s3,r186/1:/Table/94/1/{4-5}] change replicas of r186 failed: storage/replica_raft.go:305: [n3,s3,r186/1:/Table/94/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 18:32:48.686606 235383 storage/replica_command.go:1430 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r188:/Table/94/{1/6-2} [(n3,s3):1, next=2, gen=143] I190903 18:32:48.694589 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:48.696179 235383 storage/replica_raft.go:291 [n3,s3,r188/1:/Table/94/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:48.704382 235383 storage/store_snapshot.go:995 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] sending LEARNER snapshot 99c9b394 at applied index 18 I190903 18:32:48.704848 235383 storage/store_snapshot.go:1038 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:48.707503 235479 storage/replica_raftstorage.go:793 [n1,s1,r188/2:{-}] applying LEARNER snapshot [id=99c9b394 index=18] I190903 18:32:48.713736 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:48.717478 235479 storage/replica_raftstorage.go:814 [n1,s1,r188/2:/Table/94/{1/6-2}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=99c9b394 index=18] I190903 18:32:48.721660 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.726838 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.728494 235383 storage/replica_command.go:1430 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r188:/Table/94/{1/6-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=144] I190903 18:32:48.739666 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.741899 235383 storage/replica_raft.go:291 [n3,s3,r188/1:/Table/94/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:48.742459 235383 storage/replica_command.go:1430 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r188:/Table/94/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=145] I190903 18:32:48.744682 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.749857 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.754197 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.758401 235383 storage/replica_raft.go:291 [n3,s3,r188/1:/Table/94/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:48.758452 235383 storage/replica_raft.go:304 [n3,s3,r188/1:/Table/94/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:48.759331 235383 storage/replica_command.go:979 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r188 failed: storage/replica_raft.go:305: [n3,s3,r188/1:/Table/94/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:48.759479 235383 storage/queue.go:1027 [n3,replicate,s3,r188/1:/Table/94/{1/6-2}] change replicas of r188 failed: storage/replica_raft.go:305: [n3,s3,r188/1:/Table/94/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:48.761457 235586 storage/replica_command.go:1430 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r189:/Table/94/1{-/4} [(n3,s3):1, next=2, gen=141] I190903 18:32:48.761615 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:48.769061 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.774644 235586 storage/replica_raft.go:291 [n3,s3,r189/1:/Table/94/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:48.776312 235586 storage/store_snapshot.go:995 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] sending LEARNER snapshot 19e85f05 at applied index 15 I190903 18:32:48.779495 235586 storage/store_snapshot.go:1038 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:48.780834 235770 storage/replica_raftstorage.go:793 [n1,s1,r189/2:{-}] applying LEARNER snapshot [id=19e85f05 index=15] I190903 18:32:48.784808 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.793625 235770 storage/replica_raftstorage.go:814 [n1,s1,r189/2:/Table/94/1{-/4}] applied LEARNER snapshot [total=13ms ingestion=4@12ms id=19e85f05 index=15] I190903 18:32:48.794830 235586 storage/replica_command.go:1430 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r189:/Table/94/1{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=142] I190903 18:32:48.795588 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:48.802253 235586 storage/replica_raft.go:291 [n3,s3,r189/1:/Table/94/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 18:32:48.802563 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.805648 235586 storage/replica_command.go:1430 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r189:/Table/94/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:48.810741 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.819224 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:48.821236 235586 storage/replica_raft.go:291 [n3,s3,r189/1:/Table/94/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 18:32:48.821296 235586 storage/replica_raft.go:304 [n3,s3,r189/1:/Table/94/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:48.822148 235586 storage/replica_command.go:979 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] could not promote [] to voter, rolling back: change replicas of r189 failed: storage/replica_raft.go:305: [n3,s3,r189/1:/Table/94/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 18:32:48.822218 235586 storage/queue.go:1027 [n3,replicate,s3,r189/1:/Table/94/1{-/4}] change replicas of r189 failed: storage/replica_raft.go:305: [n3,s3,r189/1:/Table/94/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 18:32:48.833330 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:48.838688 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.846696 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.851380 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:48.856278 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:48.894959 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:48.900067 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.907448 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.944182 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:48.953369 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:48.983891 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:49.006164 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:49.048229 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 E190903 18:32:49.062238 236740 storage/queue.go:1027 [n2,replicate,s2,r186/2:/Table/94/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 18:32:49.126789 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:49.138549 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:49.221399 237147 storage/replica_command.go:1430 [n2,replicate,s2,r186/2:/Table/94/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r186:/Table/94/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=143] I190903 18:32:49.227555 237147 storage/replica_raft.go:291 [n2,s2,r186/2:/Table/94/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:49.235656 237126 storage/store.go:2593 [n3,replicaGC,s3,r186/1:/Table/94/1/{4-5}] removing replica r186/1 I190903 18:32:49.236074 237126 storage/replica_destroy.go:152 [n3,replicaGC,s3,r186/1:/Table/94/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:49.281841 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:49.556064 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 E190903 18:32:49.587804 238044 storage/queue.go:1027 [n1,replicate,s1,r189/2:/Table/94/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:22] E190903 18:32:49.903568 238783 storage/queue.go:1027 [n1,replicate,s1,r188/2:/Table/94/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 18:32:50.096114 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 18:32:50.100352 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 18:32:50.104663 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 18:32:50.107642 230558 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 18:32:50.115103 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=94 (unpartitioned) version=2 mtime=2019-09-03 18:32:50.109704994 +0000 UTC I190903 18:32:50.128394 239622 storage/replica_command.go:1430 [n1,replicate,s1,r189/2:/Table/94/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r189:/Table/94/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=143] I190903 18:32:50.131129 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 94 1}] I190903 18:32:50.131859 239613 sql/lease.go:923 new lease: 94("unpartitioned") ver=2:1567535879.637582614,0, refcount=0 I190903 18:32:50.136314 239622 storage/replica_raft.go:291 [n1,s1,r189/2:/Table/94/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:50.145971 240061 storage/store.go:2593 [n3,replicaGC,s3,r189/1:/Table/94/1{-/4}] removing replica r189/1 I190903 18:32:50.146346 240061 storage/replica_destroy.go:152 [n3,replicaGC,s3,r189/1:/Table/94/1{-/4}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:50.181951 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=94 (unpartitioned) version=3 mtime=2019-09-03 18:32:50.181243285 +0000 UTC I190903 18:32:50.211495 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 94 2}] I190903 18:32:50.212002 240119 sql/lease.go:923 new lease: 94("unpartitioned") ver=3:1567535879.638582614,0, refcount=0 I190903 18:32:50.237948 236032 storage/replica_command.go:1430 [n1,replicate,s1,r188/2:/Table/94/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r188:/Table/94/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=145] I190903 18:32:50.256510 236032 storage/replica_raft.go:291 [n1,s1,r188/2:/Table/94/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:50.261289 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=94 (unpartitioned) version=4 mtime=2019-09-03 18:32:50.240905598 +0000 UTC I190903 18:32:50.272613 240878 storage/store.go:2593 [n3,replicaGC,s3,r188/1:/Table/94/{1/6-2}] removing replica r188/1 I190903 18:32:50.273034 240878 storage/replica_destroy.go:152 [n3,replicaGC,s3,r188/1:/Table/94/{1/6-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:50.288936 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "alter_table", target: 94, info: {TableName:data.public.unpartitioned Statement:ALTER TABLE unpartitioned PARTITION BY NOTHING User:root MutationID:0 CascadeDroppedViews:[]} I190903 18:32:50.295827 241118 storage/replica_command.go:1430 [n3,s3,r190/1:/{Table/94/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r190:/{Table/94/2-Max} [(n3,s3):1, next=2, gen=143] I190903 18:32:50.297389 240843 storage/replica_command.go:1430 [n1,s1,r189/2:/Table/94/1{-/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r189:/Table/94/1{-/4} [(n1,s1):2, next=3, gen=144] I190903 18:32:50.297745 241003 storage/replica_command.go:1430 [n3,s3,r187/1:/Table/94/1/{5-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r187:/Table/94/1/{5-6} [(n3,s3):1, next=2, gen=142] I190903 18:32:50.306471 241118 storage/replica_raft.go:291 [n3,s3,r190/1:/{Table/94/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 18:32:50.308323 241118 storage/store_snapshot.go:995 [n3,s3,r190/1:/{Table/94/2-Max}] sending LEARNER snapshot 283a51c5 at applied index 18 I190903 18:32:50.308481 241118 storage/store_snapshot.go:1038 [n3,s3,r190/1:/{Table/94/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:50.309649 241327 storage/replica_raftstorage.go:793 [n1,s1,r190/2:{-}] applying LEARNER snapshot [id=283a51c5 index=18] I190903 18:32:50.310964 241327 storage/replica_raftstorage.go:814 [n1,s1,r190/2:/{Table/94/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=283a51c5 index=18] I190903 18:32:50.311487 241118 storage/replica_command.go:1430 [n3,s3,r190/1:/{Table/94/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r190:/{Table/94/2-Max} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=144] I190903 18:32:50.312835 241003 storage/replica_raft.go:291 [n3,s3,r187/1:/Table/94/1/{5-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 18:32:50.315416 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 94 3}] I190903 18:32:50.315539 241003 storage/store_snapshot.go:995 [n3,s3,r187/1:/Table/94/1/{5-6}] sending LEARNER snapshot 1bbc9d31 at applied index 23 I190903 18:32:50.315693 241003 storage/store_snapshot.go:1038 [n3,s3,r187/1:/Table/94/1/{5-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:50.316810 241498 storage/replica_raftstorage.go:793 [n2,s2,r187/2:{-}] applying LEARNER snapshot [id=1bbc9d31 index=23] I190903 18:32:50.319948 240843 storage/replica_raft.go:291 [n1,s1,r189/2:/Table/94/1{-/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 18:32:50.321173 240843 storage/store_snapshot.go:995 [n1,s1,r189/2:/Table/94/1{-/4}] sending LEARNER snapshot e4fc35d5 at applied index 30 I190903 18:32:50.321367 240843 storage/store_snapshot.go:1038 [n1,s1,r189/2:/Table/94/1{-/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:50.322483 240994 sql/lease.go:923 new lease: 94("unpartitioned") ver=4:1567535879.639582614,0, refcount=0 I190903 18:32:50.329484 241498 storage/replica_raftstorage.go:814 [n2,s2,r187/2:/Table/94/1/{5-6}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=1bbc9d31 index=23] I190903 18:32:50.330283 241003 storage/replica_command.go:1430 [n3,s3,r187/1:/Table/94/1/{5-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r187:/Table/94/1/{5-6} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=143] I190903 18:32:50.331964 241118 storage/replica_raft.go:291 [n3,s3,r190/1:/{Table/94/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 E190903 18:32:50.334628 241420 storage/queue.go:1027 [n1,replicate,s1,r190/2:/{Table/94/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 18:32:50.335746 241434 storage/replica_raftstorage.go:793 [n3,s3,r189/3:{-}] applying LEARNER snapshot [id=e4fc35d5 index=30] I190903 18:32:50.336131 241003 storage/replica_raft.go:291 [n3,s3,r187/1:/Table/94/1/{5-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 E190903 18:32:50.356208 241406 storage/queue.go:1027 [n2,replicate,s2,r187/2:/Table/94/1/{5-6}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 18:32:50.356952 241434 storage/replica_raftstorage.go:814 [n3,s3,r189/3:/Table/94/1{-/4}] applied LEARNER snapshot [total=21ms ingestion=4@21ms id=e4fc35d5 index=30] I190903 18:32:50.357709 240843 storage/replica_command.go:1430 [n1,s1,r189/2:/Table/94/1{-/4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r189:/Table/94/1{-/4} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=145] === RUN TestRepartitioning/single_col_range_partitioning/unpartitioned I190903 18:32:50.375512 240843 storage/replica_raft.go:291 [n1,s1,r189/2:/Table/94/1{-/4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 E190903 18:32:50.379237 241625 storage/queue.go:1027 [n3,replicate,s3,r189/3:/Table/94/1{-/4}] no removable replicas from range that needs a removal: [2:0, 3*:34] I190903 18:32:50.379290 241844 storage/replica_command.go:1430 [n3,s3,r189/3:/Table/94/1{-/4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r189:/Table/94/1{-/4} [(n1,s1):2, (n3,s3):3, next=4, gen=146] I190903 18:32:50.388766 241844 storage/replica_raft.go:291 [n3,s3,r189/3:/Table/94/1{-/4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 18:32:50.393444 241243 storage/replica_command.go:609 [n3,merge,s3,r185/1:/Table/94{-/1}] initiating a merge of r189:/Table/94/1{-/4} [(n3,s3):3, next=4, gen=147] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:50.399447 241849 storage/store.go:2593 [n1,replicaGC,s1,r189/2:/Table/94/1{-/4}] removing replica r189/2 I190903 18:32:50.399927 241849 storage/replica_destroy.go:152 [n1,replicaGC,s1,r189/2:/Table/94/1{-/4}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:50.401719 2299 storage/store.go:2593 [n3,s3,r185/1:/Table/94{-/1}] removing replica r189/3 I190903 18:32:50.413978 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root] publish: descID=94 (unpartitioned) version=5 mtime=2019-09-03 18:32:50.364821551 +0000 UTC I190903 18:32:50.439964 241773 storage/replica_command.go:1430 [n1,s1,r190/2:/{Table/94/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r190:/{Table/94/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=145] I190903 18:32:50.461013 241773 storage/replica_raft.go:291 [n1,s1,r190/2:/{Table/94/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 18:32:50.466432 241105 storage/replica_command.go:609 [n1,merge,s1,r188/2:/Table/94/{1/6-2}] initiating a merge of r190:/{Table/94/2-Max} [(n1,s1):2, next=3, gen=146] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:50.469525 241975 storage/replica_command.go:1430 [n2,s2,r187/2:/Table/94/1/{5-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r187:/Table/94/1/{5-6} [(n3,s3):1, (n2,s2):2, next=3, gen=144] I190903 18:32:50.475492 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "drop_database", target: 93, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public.unpartitioned]} I190903 18:32:50.482731 242253 storage/replica_command.go:1430 [n2,s2,r186/2:/Table/94/1/{4-5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r186:/Table/94/1/{4-5} [(n2,s2):2, next=3, gen=144] I190903 18:32:50.483646 261 storage/store.go:2593 [n1,s1,r188/2:/Table/94/{1/6-2}] removing replica r190/2 I190903 18:32:50.485261 241975 storage/replica_raft.go:291 [n2,s2,r187/2:/Table/94/1/{5-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 18:32:50.485263 1174 sql/lease.go:329 [n1,client=127.0.0.1:39262,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 94 4}] I190903 18:32:50.488300 241111 storage/replica_command.go:609 [n2,merge,s2,r186/2:/Table/94/1/{4-5}] initiating a merge of r187:/Table/94/1/{5-6} [(n2,s2):2, next=3, gen=145] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:50.488945 242253 storage/replica_raft.go:291 [n2,s2,r186/2:/Table/94/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:50.493094 242253 storage/store_snapshot.go:995 [n2,s2,r186/2:/Table/94/1/{4-5}] sending LEARNER snapshot abcb8d35 at applied index 37 I190903 18:32:50.493441 242253 storage/store_snapshot.go:1038 [n2,s2,r186/2:/Table/94/1/{4-5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:50.494460 242712 storage/replica_raftstorage.go:793 [n3,s3,r186/3:{-}] applying LEARNER snapshot [id=abcb8d35 index=37] I190903 18:32:50.495815 242712 storage/replica_raftstorage.go:814 [n3,s3,r186/3:/Table/94/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=abcb8d35 index=37] I190903 18:32:50.496296 242253 storage/replica_command.go:1430 [n2,s2,r186/2:/Table/94/1/{4-5}] change replicas (add [(n3,s3):3] remove []): existing descriptor r186:/Table/94/1/{4-5} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=145] I190903 18:32:50.499293 242788 storage/replica_command.go:1430 [n1,s1,r188/2:/{Table/94/1/6-Max}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r188:/{Table/94/1/6-Max} [(n1,s1):2, next=3, gen=147] I190903 18:32:50.502454 242253 storage/replica_raft.go:291 [n2,s2,r186/2:/Table/94/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 E190903 18:32:50.505335 242810 storage/queue.go:1027 [n3,replicate,s3,r186/3:/Table/94/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 18:32:50.506216 242788 storage/replica_raft.go:291 [n1,s1,r188/2:/{Table/94/1/6-Max}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 18:32:50.507366 242788 storage/store_snapshot.go:995 [n1,s1,r188/2:/{Table/94/1/6-Max}] sending LEARNER snapshot 7489ffd7 at applied index 39 I190903 18:32:50.507599 242788 storage/store_snapshot.go:1038 [n1,s1,r188/2:/{Table/94/1/6-Max}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 23, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 18:32:50.509205 242812 storage/replica_raftstorage.go:793 [n2,s2,r188/3:{-}] applying LEARNER snapshot [id=7489ffd7 index=39] I190903 18:32:50.512102 242812 storage/replica_raftstorage.go:814 [n2,s2,r188/3:/{Table/94/1/6-Max}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=7489ffd7 index=39] I190903 18:32:50.512933 242788 storage/replica_command.go:1430 [n1,s1,r188/2:/{Table/94/1/6-Max}] change replicas (add [(n2,s2):3] remove []): existing descriptor r188:/{Table/94/1/6-Max} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=148] I190903 18:32:50.529801 242788 storage/replica_raft.go:291 [n1,s1,r188/2:/{Table/94/1/6-Max}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 E190903 18:32:50.535538 242920 storage/queue.go:1027 [n2,replicate,s2,r188/3:/{Table/94/1/6-Max}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 18:32:50.551997 1174 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:39262,user=root,scExec] publish: descID=94 (unpartitioned) version=6 mtime=2019-09-03 18:32:50.551471163 +0000 UTC I190903 18:32:50.583294 242923 storage/replica_command.go:1430 [n2,s2,r188/3:/{Table/94/1/6-Max}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r188:/{Table/94/1/6-Max} [(n1,s1):2, (n2,s2):3, next=4, gen=149] I190903 18:32:50.590719 242923 storage/replica_raft.go:291 [n2,s2,r188/3:/{Table/94/1/6-Max}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 18:32:50.593128 242769 storage/replica_command.go:609 [n2,merge,s2,r187/2:/Table/94/1/{5-6}] initiating a merge of r188:/{Table/94/1/6-Max} [(n2,s2):3, next=4, gen=150] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) W190903 18:32:50.620016 243644 storage/intentresolver/intent_resolver.go:508 failed to push during intent resolution: failed to push id=e5878c1f key=/Local/Range/Table/94/1/5/RangeDescriptor rw=true pri=0.00715111 stat=PENDING epo=0 ts=1567535570.592949278,0 orig=0.000000000,0 min=1567535570.592949278,0 max=0.000000000,0 wto=false seq=8 I190903 18:32:50.620254 242787 storage/replica_command.go:1430 [n3,s3,r186/3:/Table/94/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r186:/Table/94/1/{4-5} [(n2,s2):2, (n3,s3):3, next=4, gen=146] I190903 18:32:50.620773 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_database", target: 95, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 18:32:50.636961 242787 storage/replica_raft.go:291 [n3,s3,r186/3:/Table/94/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 18:32:50.652821 242557 storage/replica_command.go:609 [n3,merge,s3,r185/1:/Table/94{-/1/4}] initiating a merge of r186:/Table/94/1/{4-5} [(n3,s3):3, next=4, gen=147] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 18:32:50.656216 1537 storage/store.go:2593 [n2,s2,r187/2:/Table/94/1/{5-6}] removing replica r188/3 I190903 18:32:50.659917 2338 storage/store.go:2593 [n3,s3,r185/1:/Table/94{-/1/4}] removing replica r186/3 I190903 18:32:50.668498 243610 storage/store.go:2593 [n2,replicaGC,s2,r186/2:/Table/94/1/{4-5}] removing replica r186/2 I190903 18:32:50.668577 244165 storage/replica_command.go:1430 [n2,s2,r187/2:/{Table/94/1/5-Max}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r187:/{Table/94/1/5-Max} [(n2,s2):2, next=3, gen=151] I190903 18:32:50.668919 243610 storage/replica_destroy.go:152 [n2,replicaGC,s2,r186/2:/Table/94/1/{4-5}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 18:32:50.680770 244165 storage/replica_raft.go:291 [n2,s2,r187/2:/{Table/94/1/5-Max}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 18:32:50.682048 244165 storage/store_snapshot.go:995 [n2,s2,r187/2:/{Table/94/1/5-Max}] sending LEARNER snapshot f9f0b80c at applied index 35 I190903 18:32:50.682316 244165 storage/store_snapshot.go:1038 [n2,s2,r187/2:/{Table/94/1/5-Max}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 41, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s F190903 18:32:50.683220 242722 storage/replica_raft.go:1486 [n3,s3,r187/1:/Table/94/1/{5-6}] snapshot widens existing replica, but no replica exists for subsumed key /Table/94/1/6 goroutine 242722 [running]: github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc00040e301, 0xc00040e300, 0x0, 0x6) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1016 +0xb1 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000004, 0x6bde7e6, 0x17, 0x5ce, 0xc0047ad180, 0x77) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:874 +0x93e github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e1a0, 0xc0070f9470, 0xc000000004, 0x2, 0x4038bc6, 0x4b, 0xc006408a58, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e1a0, 0xc0070f9470, 0x1, 0xc000000004, 0x4038bc6, 0x4b, 0xc006408a58, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:180 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeAcquireSnapshotMergeLock(0xc0075c9000, 0x489e1a0, 0xc0070f9470, 0xa64c41470cb8f0f9, 0xac0c1d6a1bf20093, 0xc006411fc0, 0x0, 0x0, 0x0, 0xc006db4a00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:1486 +0x294 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc0075c9000, 0x489e1a0, 0xc0070f9470, 0xa64c41470cb8f0f9, 0xac0c1d6a1bf20093, 0xc006411fc0, 0x0, 0x0, 0x0, 0xc006db4a00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:521 +0x1893 github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x489e1a0, 0xc0070f9470, 0xc0075c9000, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3488 +0x19d github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc001f42700, 0x489e1a0, 0xc0070f9470, 0xc006db47c8, 0xc0079d7418, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3368 +0x150 github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc001f42700, 0x489e1a0, 0xc0070f8ff0, 0xc006db4780, 0xa64c41470cb8f0f9, 0xac0c1d6a1bf20093, 0xc006411fc0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3430 +0x1a4 github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc001f42700, 0x489e1a0, 0xc0070f8ff0, 0xc006db4780, 0x7f08666bd1d0, 0xc0062a9450, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:871 +0x6d3 github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1(0x489e1a0, 0xc0070f8ff0, 0xc0011a2480, 0x3fbe40a) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3234 +0x1d9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0011a2480, 0x489e1a0, 0xc0070f8ff0, 0x3fbe40a, 0x1e, 0xc0079d7c30, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc001f42700, 0xc006db4780, 0x7f08666bd1a0, 0xc0062a9450, 0xc0062a9450, 0x30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3224 +0xe6 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x48fa640, 0xc0062a9450, 0xc000214960, 0x489e1a0, 0xc0070f8f30, 0x71f47f, 0xc003cf7c70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x489e1a0, 0xc0070f8f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc0070f8f30, 0xc001171780, 0x32, 0x0, 0x0, 0xc0070f8f60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1 [chan receive]: testing.(*T).Run(0xc000176c00, 0x3f999d0, 0x12, 0x40df0a8, 0x7ad001) /usr/local/go/src/testing/testing.go:917 +0x381 testing.runTests.func1(0xc000176b00) /usr/local/go/src/testing/testing.go:1157 +0x78 testing.tRunner(0xc000176b00, 0xc000835e10) /usr/local/go/src/testing/testing.go:865 +0xc0 testing.runTests(0xc0002bba80, 0x6fb55a0, 0x8, 0x8, 0x0) /usr/local/go/src/testing/testing.go:1155 +0x2a9 testing.(*M).Run(0xc0003fb900, 0x0) /usr/local/go/src/testing/testing.go:1072 +0x162 github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestMain(0xc0003fb900) /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/main_test.go:31 +0xda main.main() _testmain.go:54 +0x13e goroutine 20 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:139 +0x9c os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41 goroutine 21 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon() /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1222 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/log.init.0 /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:625 +0x121 goroutine 22 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher() /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:632 +0x126 created by github.com/cockroachdb/cockroach/pkg/util/log.init.0 /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:626 +0x139 goroutine 56 [select]: github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0000d4c00) /go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0 /go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57 goroutine 58 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000c52120, 0xc000000000) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000c52110) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc000c52000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 24 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x489e1a0, 0xc000795dd0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1716 +0x370 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5740, 0xc000cfe090, 0xc000795d70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 70 [chan receive]: testing.(*T).Run(0xc004fa9100, 0xc004a2c1b0, 0x2b, 0xc0070720c0, 0x2) /usr/local/go/src/testing/testing.go:917 +0x381 github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestRepartitioning(0xc000176c00) /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/partition_test.go:1291 +0x3a1 testing.tRunner(0xc000176c00, 0x40df0a8) /usr/local/go/src/testing/testing.go:865 +0xc0 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:916 +0x35a goroutine 71 [select]: github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc000cfe090, 0xc000cc66e0) /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x14a created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49 goroutine 72 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x489e1a0, 0xc000de8000) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:478 +0x52 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355f70, 0xc000cfe090, 0xc0003756c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 73 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x489e1a0, 0xc000dec000) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c070, 0xc000cfe090, 0xc00038c060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 59 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000866720) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ec70, 0xc000cfe090, 0xc00036ec60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 60 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000866840) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ec90, 0xc000cfe090, 0xc00036ec80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 61 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0008668a0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ecc0, 0xc000cfe090, 0xc00036eca0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 62 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000866030) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ece0, 0xc000cfe090, 0xc00036ecd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 63 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000867590) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ed10, 0xc000cfe090, 0xc00036ed00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 64 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000867530) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ed40, 0xc000cfe090, 0xc00036ed30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 65 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0008675f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ed70, 0xc000cfe090, 0xc00036ed60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 82 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000867650) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036eda0, 0xc000cfe090, 0xc00036ed90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 83 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000790120) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036edd0, 0xc000cfe090, 0xc00036edc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 84 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0008672f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ee00, 0xc000cfe090, 0xc00036ede0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 85 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000867350) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ee30, 0xc000cfe090, 0xc00036ee10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 86 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0008673b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ee60, 0xc000cfe090, 0xc00036ee40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 87 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000866ab0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ee90, 0xc000cfe090, 0xc00036ee70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 88 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000867470) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ef40, 0xc000cfe090, 0xc00036eea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 89 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000867410) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ef60, 0xc000cfe090, 0xc00036ef50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 90 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0008674d0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036ef80, 0xc000cfe090, 0xc00036ef70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 91 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc000193d40, 0x489e0e0, 0xc000384d40) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 92 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc000072240, 0x489e0e0, 0xc000384d40) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 25 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x489e1a0, 0xc000795e60) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1797 +0x257 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5760, 0xc000cfe090, 0xc0002d5750) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 26 [select]: github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x40e5438, 0xc0004addb0, 0xc0007b20e0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11b created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x105 goroutine 27 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc0005c2150) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5820, 0xc000cfe090, 0xc0003321a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 28 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x489e1a0, 0xc0005c2210) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1751 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5870, 0xc000cfe090, 0xc000332200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 29 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44d488, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000eae098, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000eae080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc000eae080, 0x726df60, 0x7f086d4986d0, 0xc000b03080) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110078, 0x71f47f, 0xc000ae02b0, 0xc000abcef0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110078, 0x40e2688, 0xc000ae0240, 0xc00026ad00, 0x0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc000ae0240, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x489e1a0, 0xc0005c3b00) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1764 +0x2f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5890, 0xc000cfe090, 0xc000332260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 30 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 net/http.(*Server).Serve(0xc000a9c5b0, 0x4886ea0, 0xc000332220, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x489e1a0, 0xc0005c3b60) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1780 +0x127 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d58a0, 0xc000cfe090, 0xc0005c2240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 31 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 crypto/tls.(*listener).Accept(0xc000332280, 0x3d2f600, 0xc0005c28d0, 0x3a88520, 0x6f95560) /usr/local/go/src/crypto/tls/tls.go:62 +0x37 net/http.(*Server).Serve(0xc000a9c410, 0x486e6e0, 0xc000332280, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x489e1a0, 0xc0005c2420) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1791 +0x3f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d58b0, 0xc000cfe090, 0xc0003322a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 32 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc0006200c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5930, 0xc000cfe090, 0xc000332340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 33 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x489e1a0, 0xc0006201b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1654 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5980, 0xc000cfe090, 0xc000332380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 98 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x489e1a0, 0xc000790150) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1706 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d59b0, 0xc000cfe090, 0xc000ae04c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 99 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc000ddc180, 0x4886ea0, 0xc0003323a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x1e9 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x489e1a0, 0xc000682000) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x46 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d59c0, 0xc000cfe090, 0xc0003323c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 100 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x489e1a0, 0xc000682030) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1198 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002d5b10, 0xc000cfe090, 0xc000620540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 357 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000cf6580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 35 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ac45c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 36 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x489e1a0, 0xc0008678c0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1228 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036e200, 0xc000cfe090, 0xc000348260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 37 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000656840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 329 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000358380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2394 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ffea40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 93 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0000e3f20, 0xc0000020dd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0000e3f10) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc0000e3e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 94 [select]: github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x489e1a0, 0xc000867a10) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:44 +0x16b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c1f0, 0xc000cfe090, 0xc000682120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 95 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x489e1a0, 0xc0008679e0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c280, 0xc000cfe090, 0xc000326200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 96 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x489e1a0, 0xc000867a40) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1365 +0x3b4 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c2a0, 0xc000cfe090, 0xc00026c290) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 97 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x489e1a0, 0xc000867ad0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1400 +0x33c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c2c0, 0xc000cfe090, 0xc00026c2b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 114 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44d3b8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000eae118, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000eae100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc000eae100, 0xc0029ba001, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110098, 0xc0000bde80, 0xc0000bde88, 0x28) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110098, 0x40e2690, 0xc000ae0480, 0x48faa00, 0xc0029ba0d0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc000ae0480, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1() /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1731 +0x2f sync.(*Once).Do(0xc00026ade0, 0xc000ea7750) /usr/local/go/src/sync/once.go:44 +0xb3 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc000867b00) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c2d0, 0xc000cfe090, 0xc000326220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 115 [semacquire]: sync.runtime_SemacquireMutex(0xc00026ade4, 0x7f086d499b00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc00026ade0) /usr/local/go/src/sync/mutex.go:134 +0x109 sync.(*Once).Do(0xc00026ade0, 0xc000ea7f50) /usr/local/go/src/sync/once.go:40 +0x3b github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc000de8090) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c510, 0xc000cfe090, 0xc000326f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 116 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc00083e5b0, 0x489e1a0, 0xc000734000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:134 +0x333 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c730, 0xc000cfe090, 0xc00026c720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 130 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc000734060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c820, 0xc000cfe090, 0xc00030c020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 117 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0002f2000, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc00007aa80, 0x22, 0x0, 0x0, 0xc00026c980) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 118 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0002f29a0, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc00007ab40, 0x22, 0x0, 0x0, 0xc00026c990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 119 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x489e1a0, 0xc000de8420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:151 +0x4a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d400, 0xc000cfe090, 0xc000301300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 120 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6918) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de8450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de8450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d430, 0xc000cfe090, 0xc00026d410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 121 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6984) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de8480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de8480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d450, 0xc000cfe090, 0xc00026d440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 122 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6991) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de84b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de84b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d480, 0xc000cfe090, 0xc00026d470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 123 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x697f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de84e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de84e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d4b0, 0xc000cfe090, 0xc00026d490) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 124 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x698b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de8ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de8ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d4d0, 0xc000cfe090, 0xc00026d4c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 125 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6961) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de8270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de8270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d500, 0xc000cfe090, 0xc00026d4f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 126 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x697e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de8f90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de8f90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d530, 0xc000cfe090, 0xc00026d510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 127 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6978) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d550, 0xc000cfe090, 0xc00026d540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 128 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x691c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de8f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de8f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d580, 0xc000cfe090, 0xc00026d570) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 129 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6977) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de90b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de90b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d5b0, 0xc000cfe090, 0xc00026d590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 146 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6928) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d5d0, 0xc000cfe090, 0xc00026d5c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 147 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6980) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de90e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de90e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d600, 0xc000cfe090, 0xc00026d5f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 148 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x696d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d630, 0xc000cfe090, 0xc00026d610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 149 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6981) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007340f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007340f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d650, 0xc000cfe090, 0xc00026d640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 150 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6988) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d670, 0xc000cfe090, 0xc00026d660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 151 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x698d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d690, 0xc000cfe090, 0xc00026d680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 152 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6990) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007341b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007341b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d6b0, 0xc000cfe090, 0xc00026d6a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 153 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6931) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007341e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007341e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d6e0, 0xc000cfe090, 0xc00026d6d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 154 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6979) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d710, 0xc000cfe090, 0xc00026d6f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 155 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x697c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d740, 0xc000cfe090, 0xc00026d720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 156 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6993) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d770, 0xc000cfe090, 0xc00026d750) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 157 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6926) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007342a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007342a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d790, 0xc000cfe090, 0xc00026d780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 158 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6989) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d7c0, 0xc000cfe090, 0xc00026d7b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 159 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6930) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d7f0, 0xc000cfe090, 0xc00026d7e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 160 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6924) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d860, 0xc000cfe090, 0xc00026d850) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 161 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6927) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d880, 0xc000cfe090, 0xc00026d870) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 162 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x691d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007343c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007343c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d8a0, 0xc000cfe090, 0xc00026d890) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 163 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x697d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007343f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007343f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d8c0, 0xc000cfe090, 0xc00026d8b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 164 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6997) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d8e0, 0xc000cfe090, 0xc00026d8d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 165 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6923) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d900, 0xc000cfe090, 0xc00026d8f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 166 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6996) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d920, 0xc000cfe090, 0xc00026d910) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 167 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x691a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007344b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007344b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d940, 0xc000cfe090, 0xc00026d930) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 168 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6982) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007340c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007340c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d960, 0xc000cfe090, 0xc00026d950) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 169 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x691f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d980, 0xc000cfe090, 0xc00026d970) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 170 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6925) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de91a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de91a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d9a0, 0xc000cfe090, 0xc00026d990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 171 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x693b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de91d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de91d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d9c0, 0xc000cfe090, 0xc00026d9b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 172 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x692c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026d9e0, 0xc000cfe090, 0xc00026d9d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 173 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x692e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026da00, 0xc000cfe090, 0xc00026d9f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 174 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6921) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026da20, 0xc000cfe090, 0xc00026da10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 175 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6920) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026da40, 0xc000cfe090, 0xc00026da30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 176 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x691e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007345d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007345d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026da60, 0xc000cfe090, 0xc00026da50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 177 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6944) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de92c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de92c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026da80, 0xc000cfe090, 0xc00026da70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 178 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6948) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026daa0, 0xc000cfe090, 0xc00026da90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 179 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6945) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dac0, 0xc000cfe090, 0xc00026dab0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 180 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x696f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de92f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de92f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dae0, 0xc000cfe090, 0xc00026dad0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 181 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x692f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026db00, 0xc000cfe090, 0xc00026daf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 182 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x698c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026db20, 0xc000cfe090, 0xc00026db10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 183 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6994) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007344e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007344e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026db40, 0xc000cfe090, 0xc00026db30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 184 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x693c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc0007345a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007345a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026db60, 0xc000cfe090, 0xc00026db50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 185 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6936) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026db90, 0xc000cfe090, 0xc00026db80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 186 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x691b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000734600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000734600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dbc0, 0xc000cfe090, 0xc00026dba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 187 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6933) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dbf0, 0xc000cfe090, 0xc00026dbd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 188 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6943) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dc20, 0xc000cfe090, 0xc00026dc00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 189 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6929) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dc40, 0xc000cfe090, 0xc00026dc30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 190 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x693e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dc70, 0xc000cfe090, 0xc00026dc60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 191 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6922) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de93e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de93e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dca0, 0xc000cfe090, 0xc00026dc90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 192 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x694d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dd10, 0xc000cfe090, 0xc00026dd00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 193 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6934) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de93b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de93b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dd30, 0xc000cfe090, 0xc00026dd20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 194 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x692b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dd50, 0xc000cfe090, 0xc00026dd40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 195 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6964) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de94a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de94a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dd70, 0xc000cfe090, 0xc00026dd60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 196 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6919) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dd90, 0xc000cfe090, 0xc00026dd80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 197 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x696e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026ddb0, 0xc000cfe090, 0xc00026dda0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 198 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6937) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026ddd0, 0xc000cfe090, 0xc00026ddc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 199 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6950) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026ddf0, 0xc000cfe090, 0xc00026dde0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 200 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6983) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de94d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de94d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026de10, 0xc000cfe090, 0xc00026de00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 201 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6940) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de95c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de95c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026de30, 0xc000cfe090, 0xc00026de20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 202 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x693a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de95f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de95f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026de50, 0xc000cfe090, 0xc00026de40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 203 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x692d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026de70, 0xc000cfe090, 0xc00026de60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 204 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x692a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026de90, 0xc000cfe090, 0xc00026de80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 205 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6939) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026deb0, 0xc000cfe090, 0xc00026dea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 206 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x695e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de96b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de96b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026ded0, 0xc000cfe090, 0xc00026dec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 207 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x694e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de96e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de96e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026def0, 0xc000cfe090, 0xc00026dee0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 208 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x695b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026df10, 0xc000cfe090, 0xc00026df00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 209 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6954) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026df30, 0xc000cfe090, 0xc00026df20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 210 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6951) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026df50, 0xc000cfe090, 0xc00026df40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 211 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x694b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de97a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de97a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026df70, 0xc000cfe090, 0xc00026df60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 212 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6949) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de97d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de97d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026df90, 0xc000cfe090, 0xc00026df80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 213 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6932) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dfb0, 0xc000cfe090, 0xc00026dfa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 214 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6985) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026dfe0, 0xc000cfe090, 0xc00026dfd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 215 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6941) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216000, 0xc000cfe090, 0xc00026dff0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 216 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6955) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216050, 0xc000cfe090, 0xc000216030) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 217 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6935) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de98c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de98c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002162d0, 0xc000cfe090, 0xc0002162c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 218 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6967) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de98f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de98f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216300, 0xc000cfe090, 0xc0002162e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 219 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6947) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216320, 0xc000cfe090, 0xc000216310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 220 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x694a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216340, 0xc000cfe090, 0xc000216330) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 221 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6938) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216380, 0xc000cfe090, 0xc000216370) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 222 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6946) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de99b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de99b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002163a0, 0xc000cfe090, 0xc000216390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 223 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6960) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de99e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de99e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002163c0, 0xc000cfe090, 0xc0002163b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 224 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6959) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002163f0, 0xc000cfe090, 0xc0002163e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 225 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x693d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216420, 0xc000cfe090, 0xc000216400) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 226 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x694c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216450, 0xc000cfe090, 0xc000216430) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 227 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x695c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216480, 0xc000cfe090, 0xc000216460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 228 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6963) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002164c0, 0xc000cfe090, 0xc0002164a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 229 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6956) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216500, 0xc000cfe090, 0xc0002164d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 230 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x693f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216570, 0xc000cfe090, 0xc000216560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 231 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x697b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002165a0, 0xc000cfe090, 0xc000216590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 232 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6976) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002165e0, 0xc000cfe090, 0xc0002165c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 233 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6942) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216600, 0xc000cfe090, 0xc0002165f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 234 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x696b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216630, 0xc000cfe090, 0xc000216620) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 235 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6957) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216660, 0xc000cfe090, 0xc000216640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 236 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6952) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216680, 0xc000cfe090, 0xc000216670) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 237 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x696c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002166b0, 0xc000cfe090, 0xc0002166a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 238 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x694f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002166e0, 0xc000cfe090, 0xc0002166c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 239 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6962) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216700, 0xc000cfe090, 0xc0002166f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 240 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6953) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216730, 0xc000cfe090, 0xc000216720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 241 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6966) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216750, 0xc000cfe090, 0xc000216740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x698a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216770, 0xc000cfe090, 0xc000216760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 243 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x698e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002167c0, 0xc000cfe090, 0xc000216780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 244 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x695f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216800, 0xc000cfe090, 0xc0002167d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 245 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6958) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216830, 0xc000cfe090, 0xc000216820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 246 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6986) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216880, 0xc000cfe090, 0xc000216860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 247 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6965) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002168c0, 0xc000cfe090, 0xc000216890) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 248 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x697a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002168f0, 0xc000cfe090, 0xc0002168d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 249 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x695a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216920, 0xc000cfe090, 0xc000216900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 250 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x695d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216950, 0xc000cfe090, 0xc000216930) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 251 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6971) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216980, 0xc000cfe090, 0xc000216960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 252 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6973) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216a00, 0xc000cfe090, 0xc0002169d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 253 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6972) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216a30, 0xc000cfe090, 0xc000216a10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 254 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6975) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216a50, 0xc000cfe090, 0xc000216a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 255 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6968) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216a80, 0xc000cfe090, 0xc000216a70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 256 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6992) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216ab0, 0xc000cfe090, 0xc000216a90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 257 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x698f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216ad0, 0xc000cfe090, 0xc000216ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 258 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x696a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867e60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867e60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216b00, 0xc000cfe090, 0xc000216af0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 259 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6987) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216b30, 0xc000cfe090, 0xc000216b10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 260 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6974) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216b50, 0xc000cfe090, 0xc000216b40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 261 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6969) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000de9e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000de9e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216b80, 0xc000cfe090, 0xc000216b70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 262 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6995) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216bb0, 0xc000cfe090, 0xc000216b90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 263 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4ed0, 0x6970) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007d880, 0x489e1a0, 0xc000867e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000867e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216bd0, 0xc000cfe090, 0xc000216bc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 264 [semacquire]: sync.runtime_Semacquire(0xc00007d8f8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc00007d8f0) /usr/local/go/src/sync/waitgroup.go:130 +0x65 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216bf0, 0xc000cfe090, 0xc000216be0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 265 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc000ce8000, 0x489e1a0, 0xc000de9e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3769 +0x264 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216c10, 0xc000cfe090, 0xc000216c00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 266 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc000ce8000, 0x489e1a0, 0xc000867ef0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3825 +0x129 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216c30, 0xc000cfe090, 0xc000216c20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 267 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x489e1a0, 0xc000867f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1417 +0xe6 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216c90, 0xc000cfe090, 0xc000301320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 268 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000867f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216cb0, 0xc000cfe090, 0xc000b86000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 269 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000867fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216cc0, 0xc000cfe090, 0xc000b86050) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 270 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000de9ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216ce0, 0xc000cfe090, 0xc000b860a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1262 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x489e1a0, 0xc0016b7f80) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1797 +0x257 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8050, 0xc00108b320, 0xc0013f8040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 272 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x489e1a0, 0xc000bc2090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x2ea github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216d40, 0xc000cfe090, 0xc000216d20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 273 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4e10, 0xa4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4e00) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc00083e5b0, 0x489e1a0, 0xc000790e40, 0xc000cba3c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x489e1a0, 0xc000790e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1628 +0x57 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e1a0, 0xc000790e40, 0xc000230090, 0x25, 0x0, 0x0, 0xc000301340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 274 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x489e1a0, 0xc000867f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1636 +0x2db github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216d50, 0xc000cfe090, 0xc000301380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 275 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x489e1a0, 0xc000734630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:182 +0x181 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216d60, 0xc000cfe090, 0xc0003013a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 276 [select]: github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x489e1a0, 0xc000bc20c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:123 +0x148 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000216d80, 0xc000cfe090, 0xc0003013c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 81 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x489e1a0, 0xc000c373b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:682 +0x234 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000284710, 0xc000cfe090, 0xc000314a20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1332 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0018b0900, 0x489e0e0, 0xc0010df6c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 79 [select]: github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x489e1a0, 0xc000c37320) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:147 +0x14f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002846f0, 0xc000cfe090, 0xc0002846e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 132 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0007346c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8a0, 0xc000cfe090, 0xc00030c140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 133 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0007346f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8c0, 0xc000cfe090, 0xc00030c160) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 134 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000734720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8d0, 0xc000cfe090, 0xc00030c180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 135 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000734750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c900, 0xc000cfe090, 0xc00030c1a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 136 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0007347b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c910, 0xc000cfe090, 0xc00030c1e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 137 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0007347e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c930, 0xc000cfe090, 0xc00030c200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 138 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000734810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c960, 0xc000cfe090, 0xc00030c220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 139 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000734840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c970, 0xc000cfe090, 0xc00030c240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 140 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000734870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c9d0, 0xc000cfe090, 0xc00030c260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 141 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc000a9d6c0, 0x489e1a0, 0xc000734690, 0xc000cfe090, 0x2440f965, 0xed500a8d2, 0x0, 0xc000d46000, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:217 +0x1d1 github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc000d46000, 0xc000790901) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:279 +0x81 github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc0007909c0, 0xc004180b80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x18d github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x489e1a0, 0xc000734690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:277 +0x19b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c9e0, 0xc000cfe090, 0xc00030c280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 322 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x489e1a0, 0xc000cd2270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:488 +0x479 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000284840, 0xc000cfe090, 0xc000c729b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 80 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x489e1a0, 0xc000c37380) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:324 +0xfa github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000284700, 0xc000cfe090, 0xc000314a00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 277 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x489e1a0, 0xc000c78000) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217190, 0xc000cfe090, 0xc000301660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 278 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x489e1a0, 0xc000bc33b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217240, 0xc000cfe090, 0xc0003016a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 279 [select]: github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc0004addb0, 0xc0007b3e30, 0xc000217340, 0x40e5438, 0xc000201c80, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1014 +0x18b github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x489e1a0, 0xc000c37020) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217350, 0xc000cfe090, 0xc000b91980) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 280 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x489e1a0, 0xc000bc3410) /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x107 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217360, 0xc000cfe090, 0xc000301740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 281 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x489e1a0, 0xc000c37050) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2024 +0x1fc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217450, 0xc000cfe090, 0xc000b91a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 282 [select]: github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x489e1a0, 0xc000bc3440) /go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x112 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217470, 0xc000cfe090, 0xc000217460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 283 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x489e1a0, 0xc000735da0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1991 +0x80e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217480, 0xc000cfe090, 0xc000301820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1297 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000deca20) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f6b0, 0xc00108b320, 0xc00102f6a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1331 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000decae0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f6f0, 0xc00108b320, 0xc00102f6e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2496 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214870, 0x3, 0xc002455140, 0xc001171a00, 0x48fa520, 0xc00026c060, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc0007919e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000303f30, 0xc000cfe090, 0xc000791980) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1495 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44c858, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001502a98, 0x72, 0x2700, 0x2730, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001502a80, 0xc00126e000, 0x2730, 0x2730, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001502a80, 0xc00126e000, 0x2730, 0x2730, 0xc000812938, 0x2701, 0xc00176bd80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110480, 0xc00126e000, 0x2730, 0x2730, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc000cad320, 0xc00126e000, 0x2730, 0x2730, 0xc00126e00d, 0xe, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc001f64f60, 0xc00126e000, 0x2730, 0x2730, 0x2b, 0x482c600, 0xc000812a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00176be58, 0x482c340, 0xc001f64f60, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00176bc00, 0x7f086d36ae18, 0xc000cad320, 0x5, 0xc000cad320, 0xe) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00176bc00, 0x40e5100, 0xc00176bd38, 0x4b) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00176bc00, 0xc00120e000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc00133ab40, 0xc00152aab8, 0x9, 0x9, 0xc002d64388, 0xc003040d40, 0x5) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc00133ab40, 0xc00152aab8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00152aab8, 0x9, 0x9, 0x482c140, 0xc00133ab40, 0x0, 0x4b00000000, 0xc002dce100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00152aa80, 0xc000c376b0, 0xc000c376b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000fae900, 0xc000b0ca50, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc000fae900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc000fae900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2153 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002e3a580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 560 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44d2e8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000eae198, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000eae180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc000eae180, 0xc000bcc001, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc0001100a0, 0xc000eb7e68, 0xc000eb7e70, 0x38) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc0001100a0, 0x40e2110, 0xc000cfe090, 0x489e1a0, 0xc000dec210) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc000c74b20, 0x489e1a0, 0xc000dec210, 0xc000cfe090, 0x4886160, 0xc0001100a0, 0xc000dec240, 0xc000dec210, 0xc000b7df68) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x489e1a0, 0xc000dec210) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1809 +0xdd github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000334530, 0xc000cfe090, 0xc000c37a10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6746 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453b30, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc002852098, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc002852080, 0xc000b0f800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc002852080, 0xc000b0f800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc004b940a8, 0xc000b0f800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc007a90080, 0xc000b0f800, 0x791, 0x791, 0x42, 0x482c600, 0xc00336da78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001384cd8, 0x482c340, 0xc007a90080, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001384a80, 0x48317e0, 0xc004b940a8, 0x5, 0xc004b940a8, 0xc006866d50) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001384a80, 0x40e5100, 0xc001384bb8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001384a80, 0xc002c4a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003d3a300, 0xc00171e3b8, 0x9, 0x9, 0xc0050b8118, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003d3a300, 0xc00171e3b8, 0x9, 0x9, 0x9, 0xc0054d0480, 0x485e320, 0xc005504510) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00171e3b8, 0x9, 0x9, 0x482c140, 0xc003d3a300, 0x0, 0xc000000000, 0xc0069b2f00, 0x6866d00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00171e380, 0xc006866d20, 0xc006866d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0032be1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 284 [select]: net.(*pipe).read(0xc000eae300, 0xc000598000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc000eae300, 0xc000598000, 0x791, 0x791, 0xc00059800d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc00637e740, 0xc000598000, 0x791, 0x791, 0x2e, 0x482c600, 0xc001500ac0) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000b01058, 0x482c340, 0xc00637e740, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000b00e00, 0x7f086d451a50, 0xc000eae300, 0x5, 0xc000eae300, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000b00e00, 0x40e5100, 0xc000b00f38, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000b00e00, 0xc000fa0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000ee66c0, 0xc0002f2b98, 0x9, 0x9, 0x6fe9a9, 0xc000fae300, 0xc001500e40) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000ee66c0, 0xc0002f2b98, 0x9, 0x9, 0x9, 0x0, 0xc0006371f8, 0x74a6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002f2b98, 0x9, 0x9, 0x482c140, 0xc000ee66c0, 0x0, 0xc000000000, 0xc001500ea7, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002f2b60, 0xc0030ec140, 0xc0030ec140, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000c04380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1421 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00149db00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1395 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000fc7da0, 0xc000001cc1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000fc7d90) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc000fc7c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 1379 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 net/http.(*Server).Serve(0xc0017e7110, 0x4886ea0, 0xc0003bfe00, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x489e1a0, 0xc0016b7fb0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1780 +0x127 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8150, 0xc00108b320, 0xc0011efc20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1151 [select]: database/sql.(*DB).connectionOpener(0xc000bfe300, 0x489e0e0, 0xc0013d9e80) /usr/local/go/src/database/sql/sql.go:1000 +0xe8 created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:670 +0x15e goroutine 2383 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0018d2c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1152 [select]: database/sql.(*DB).connectionResetter(0xc000bfe300, 0x489e0e0, 0xc0013d9e80) /usr/local/go/src/database/sql/sql.go:1013 +0xfb created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:671 +0x194 goroutine 285 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000ee5a00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000cba8a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000c04380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 286 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000ae0a00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000cc5c20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00042af00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 287 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00042af00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 288 [select]: net.(*pipe).read(0xc000eae280, 0xc000598800, 0x75b, 0x75b, 0x203001, 0x0, 0x74e) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc000eae280, 0xc000598800, 0x75b, 0x75b, 0xc00059880d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc00637e760, 0xc000598800, 0x75b, 0x75b, 0x2e, 0x482c600, 0xc000ac8a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000ede5d8, 0x482c340, 0xc00637e760, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000ede380, 0x7f086d451a50, 0xc000eae280, 0x5, 0xc000eae280, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000ede380, 0x40e5100, 0xc000ede4b8, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000ede380, 0xc000f84000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000ee65a0, 0xc0002188f8, 0x9, 0x9, 0xc000b00000, 0x7f086d499b28, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000ee65a0, 0xc0002188f8, 0x9, 0x9, 0x9, 0xbdd055, 0xc0030ec16c, 0xc003e1df00) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002188f8, 0x9, 0x9, 0x482c140, 0xc000ee65a0, 0x0, 0x0, 0xc0030ec160, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002188c0, 0xc0030ec160, 0xc0030ec160, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00042af00, 0xc000c373e0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc00042af00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc00042af00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 356 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000cf1040, 0xc000c2c360, 0xc000298010, 0xf, 0xc001259560, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000c78210) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003541f0, 0xc000cfe090, 0xc000fa8140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1295 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec9f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f670, 0xc00108b320, 0xc00102f660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 8444 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0011ac380, 0xc004672000, 0x489e1a0, 0xc0010f5560) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 6747 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003b36200, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003d3a3c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0032be1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 355 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001368440) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 432 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x489e1a0, 0xc001239980) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:308 +0xda github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003557f0, 0xc000cfe090, 0xc0002a1580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242545 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc003efe8a0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0x6, 0xc005cc1560, 0xc002c28738, 0xc00524d700) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc004798400, 0x33, 0x0, 0x0, 0xc006bb4cc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1334 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x489e1a0, 0xc000b581b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1716 +0x370 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fb00, 0xc00108b320, 0xc0013d5ec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1009 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0017d6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1330 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000decab0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f6d0, 0xc00108b320, 0xc00102f6c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 394 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44d148, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000d26098, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000d26080, 0xc000599000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000d26080, 0xc000599000, 0x75b, 0x75b, 0xc0014e9938, 0x701, 0x294a5294a5) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000c1a010, 0xc000599000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0012bd920, 0xc000599000, 0x75b, 0x75b, 0x270, 0x280, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0038483e0, 0xc000599000, 0x75b, 0x75b, 0x203001, 0x7370f2, 0xc0014e9a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000640258, 0x482c340, 0xc0038483e0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000640000, 0x7f086d36ae18, 0xc0012bd920, 0x5, 0xc0012bd920, 0x0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000640000, 0x40e5100, 0xc000640138, 0xc004c027e0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000640000, 0xc0015cc000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000c15560, 0xc000219df8, 0x9, 0x9, 0xc00602c048, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000c15560, 0xc000219df8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000219df8, 0x9, 0x9, 0x482c140, 0xc000c15560, 0x0, 0x5500000000, 0xc003bba800, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000219dc0, 0xc000c367b0, 0xc000c367b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0012c8000, 0xc000b59680, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc0012c8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc0012c8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 431 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x489e1a0, 0xc000ffc870) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:297 +0x12b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003557e0, 0xc000cfe090, 0xc000b0d1a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1296 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec9c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f690, 0xc00108b320, 0xc00102f680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1333 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0018b0a20, 0x489e0e0, 0xc0010df6c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 392 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000c9bc00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000748d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0012c8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 393 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0012c8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 244051 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xb9, 0x490bae0, 0xc006cd51a0, 0x0, 0x1, 0x3e8e720, 0x6) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xb9, 0xc004a9a000, 0xc005b68800, 0x15c1014bdcf3449a) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc00645f200, 0x33, 0x0, 0x0, 0xc006cdb110) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2887 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc001114960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0011592c0, 0xc0011a2480, 0xc001760fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1394 [select]: net.(*pipe).read(0xc00140e180, 0xc0015f9800, 0x75b, 0x75b, 0x203001, 0x0, 0x74e) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc00140e180, 0xc0015f9800, 0x75b, 0x75b, 0xc0015f980d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc007557d00, 0xc0015f9800, 0x75b, 0x75b, 0x2e, 0x482c600, 0xc000beea50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001d1e258, 0x482c340, 0xc007557d00, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001d1e000, 0x7f086d451a50, 0xc00140e180, 0x5, 0xc00140e180, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001d1e000, 0x40e5100, 0xc001d1e138, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001d1e000, 0xc000f38000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc00192c0c0, 0xc0002f3298, 0x9, 0x9, 0xc0018d7180, 0x7f086d4acde0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc00192c0c0, 0xc0002f3298, 0x9, 0x9, 0x9, 0xbdd055, 0xc006aefc8c, 0xc003bf2060) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002f3298, 0x9, 0x9, 0x482c140, 0xc00192c0c0, 0x0, 0x0, 0xc006aefc80, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002f3260, 0xc006aefc80, 0xc006aefc80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0017d6000, 0xc0016b6270, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc001a74c00, 0x48fef60, 0xc0017d6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc001a74c00, 0x48fef60, 0xc0017d6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 244776 [semacquire]: sync.runtime_SemacquireMutex(0x726cb6c, 0xca6dc5800) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0x726cb68) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000001, 0x6bde2b1, 0x1a, 0x11c, 0xc002334000, 0xad) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:857 +0x206 github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e160, 0xc007bd6e40, 0xc000000001, 0x2, 0x401821d, 0x37, 0xc007a137d0, 0x4, 0x4) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e160, 0xc007bd6e40, 0x1, 0x1, 0x401821d, 0x37, 0xc007a137d0, 0x4, 0x4) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Infof(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:95 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor(0xc007611800, 0x489e160, 0xc007bd6e40, 0xc005e5a9a0, 0x1, 0x8, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:284 +0xf2d github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).processAttempt(0xc0002f1d80, 0x489e160, 0xc007bd6e40, 0xc007611800, 0xc004d26cd0, 0xd, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:163 +0x196 github.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).process(0xc0002f1d80, 0x489e160, 0xc007bd6e40, 0xc007611800, 0xc004d26cd0, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:141 +0x5d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc007bd6e40, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc007bd6e40, 0xc00149f740, 0x1f, 0xdf8475800, 0xc006a4f1a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc000ff3b80, 0x489e1a0, 0xc006a4f170, 0x4906de0, 0xc007611800, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc006a4f110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e1a0, 0xc006a4f110, 0xc004377560, 0x29, 0x0, 0x0, 0xc006a4f140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2696 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000fff5c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001752f00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0032ca300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2915 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4e10, 0xa3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4e00) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc00083e5b0, 0x489e1a0, 0xc001108d50, 0xc0029b6600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc001108d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e1a0, 0xc001108d50, 0xc0002320e0, 0x1d, 0x0, 0x0, 0xc0002c45a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2834 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001290700, 0xc0017ae000, 0x489e1a0, 0xc000b586c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2742 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc002cb97c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2743 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0016ad380, 0xc0014536e0, 0xc000368270, 0xf, 0xc00170e3c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000bc27e0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106dc10, 0xc00108b320, 0xc002e55950) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2744 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0011202c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2777 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0010281c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2172 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001029500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1007 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000e4e200, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00192c180, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000358a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 498 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x489e1a0, 0xc000ffd1a0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:246 +0x193 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355810, 0xc000cfe090, 0xc0002a15a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 433 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x489e1a0, 0xc0012399e0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:322 +0xfb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355800, 0xc000cfe090, 0xc000b0d1d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1293 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec990) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f630, 0xc00108b320, 0xc00102f620) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 430 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x489e1a0, 0xc0012399b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003557d0, 0xc000cfe090, 0xc0002a1560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1398 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x489e1a0, 0xc0016b65d0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1365 +0x3b4 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c200, 0xc00108b320, 0xc00106c1f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 558 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x489e1a0, 0xc000dec5a0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:171 +0x536 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003344d0, 0xc000cfe090, 0xc000361240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 330 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44d218, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000eae498, 0x72, 0x2500, 0x2550, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000eae480, 0xc00272c000, 0x2550, 0x2550, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000eae480, 0xc00272c000, 0x2550, 0x2550, 0x203001, 0x0, 0x2543) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcd608, 0xc00272c000, 0x2550, 0x2550, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0064d6240, 0xc00272c000, 0x2550, 0x2550, 0x42, 0x482c600, 0xc001229a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00065fad8, 0x482c340, 0xc0064d6240, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00065f880, 0x48317e0, 0xc000bcd608, 0x5, 0xc000bcd608, 0xc00588f350) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00065f880, 0x40e5100, 0xc00065f9b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00065f880, 0xc0015b0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0012bdce0, 0xc00152b298, 0x9, 0x9, 0xc0054aa898, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0012bdce0, 0xc00152b298, 0x9, 0x9, 0x9, 0xc006997410, 0x485e320, 0xc007bffd50) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00152b298, 0x9, 0x9, 0x482c140, 0xc0012bdce0, 0x0, 0xc000000000, 0xc004d3b100, 0x588f100) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00152b260, 0xc00588f110, 0xc00588f110, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000358380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 331 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00154d640, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0012bdec0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000358380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2371 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8c0, 0xc0011a2480, 0xc0016be080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1422 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44c9f8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001ae2018, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001ae2000, 0xc0010f3000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001ae2000, 0xc0010f3000, 0x75b, 0x75b, 0xc00177e938, 0x701, 0xc001772180) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000010040, 0xc0010f3000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc000203440, 0xc0010f3000, 0x75b, 0x75b, 0xc0010f300d, 0x83, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0075e7740, 0xc0010f3000, 0x75b, 0x75b, 0xa0, 0x482c600, 0xc00177ea50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001772258, 0x482c340, 0xc0075e7740, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001772000, 0x7f086d36ae18, 0xc000203440, 0x5, 0xc000203440, 0x83) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001772000, 0x40e5100, 0xc001772138, 0xb) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001772000, 0xc001174000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000c2a420, 0xc0003678b8, 0x9, 0x9, 0xc003b37388, 0xc00064c500, 0x7a) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000c2a420, 0xc0003678b8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003678b8, 0x9, 0x9, 0x482c140, 0xc000c2a420, 0x0, 0xb00000000, 0xc003944800, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000367880, 0xc000b90600, 0xc000b90600, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00149db00, 0xc000b90de0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc00149db00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc00149db00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1325 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc00142c150) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00172e660, 0xc00108b320, 0xc001699720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1283 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001a75020, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001a75010) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc001a74f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 2080 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c860, 0xc0011a2480, 0xc0016be020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1286 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec840) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f550, 0xc00108b320, 0xc00102f540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1294 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec8d0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f650, 0xc00108b320, 0xc00102f640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 650 [select]: database/sql.(*DB).connectionOpener(0xc0011b8180, 0x489e0e0, 0xc000ee5940) /usr/local/go/src/database/sql/sql.go:1000 +0xe8 created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:670 +0x15e goroutine 1282 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x489e1a0, 0xc0010302a0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e560, 0xc00108b320, 0xc00102e550) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1488 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00152b420, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc000321740, 0x22, 0x0, 0x0, 0xc0003353b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1506 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335400, 0xc00108b320, 0xc0003353f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1705 [select]: github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc0011a2480, 0xc001af46e0) /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x14a created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49 goroutine 2384 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a0f8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc002dc4698, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc002dc4680, 0xc0028d8800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc002dc4680, 0xc0028d8800, 0x75b, 0x75b, 0xc00277f938, 0x701, 0xc002e58c00) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00043c2f0, 0xc0028d8800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002d6fe60, 0xc0028d8800, 0x75b, 0x75b, 0xc0028d880d, 0x65, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc007d74580, 0xc0028d8800, 0x75b, 0x75b, 0x82, 0x482c600, 0xc00277fa50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002e58cd8, 0x482c340, 0xc007d74580, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002e58a80, 0x7f086d36ae18, 0xc002d6fe60, 0x5, 0xc002d6fe60, 0x65) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002e58a80, 0x40e5100, 0xc002e58bb8, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002e58a80, 0xc001ede000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc00295fc20, 0xc00295a658, 0x9, 0x9, 0xc001571d08, 0xc003af38c0, 0x5c) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc00295fc20, 0xc00295a658, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00295a658, 0x9, 0x9, 0x482c140, 0xc00295fc20, 0x0, 0x300000000, 0xc0015c4600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00295a620, 0xc0010f4030, 0xc0010f4030, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0018d2c00, 0xc0010f4180, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc001a74c00, 0x48fef60, 0xc0018d2c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc001a74c00, 0x48fef60, 0xc0018d2c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1683 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44c6b8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0017a6498, 0x72, 0x700, 0x719, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0017a6480, 0xc00168c800, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0017a6480, 0xc00168c800, 0x719, 0x719, 0x203001, 0x48314e0, 0x72b4ea0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000c74040, 0xc00168c800, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc006478760, 0xc00168c800, 0x719, 0x719, 0x0, 0x48317a0, 0xc0014e5618) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0010785d8, 0x482c340, 0xc006478760, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001078380, 0x48317e0, 0xc000c74040, 0x5, 0xc000c74040, 0xc0066e5400) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001078380, 0x40e5100, 0xc0010784b8, 0xed500a8d3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001078380, 0xc0023eb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc0002e67c0, 0xc0023eb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1896 +0x110 bufio.(*Reader).fill(0xc000143648) /usr/local/go/src/bufio/bufio.go:100 +0x10f bufio.(*Reader).ReadByte(0xc000143648, 0x1, 0x1, 0x2) /usr/local/go/src/bufio/bufio.go:249 +0x39 github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc000143da0, 0x486e520, 0xc000143648, 0x4860120, 0x72b56a8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc000143600, 0x489e0e0, 0xc000e4e880, 0xc00112e120, 0xc001324500, 0x5400, 0x15000, 0xc001660198, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:313 +0x474 github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x489e0e0, 0xc000e4e880, 0x48f9e60, 0xc001078380, 0xc00112d00d, 0x4, 0xc0013f12f0, 0x485eca0, 0xc001a57770, 0x4000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:168 +0x18d github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc001660000, 0x489e0e0, 0xc000e4e880, 0x48f9e60, 0xc001078380, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:536 +0x87e github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x48faa00, 0xc000c74040) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1813 +0x165 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc00108b320, 0x489e1a0, 0xc0015c6ab0, 0xc000c1a278, 0x48faa00, 0xc000c74040, 0xc0015c6ae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xa9 created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b goroutine 1008 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0010ec2c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00192c1e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0017d6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1148 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44cd38, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00140e018, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc00140e000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc00140e000, 0xc000c74001, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110718, 0xc000813e68, 0xc000813e70, 0x38) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110718, 0x40e2110, 0xc00108b320, 0x489e1a0, 0xc0015c6ab0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc000c1a278, 0x489e1a0, 0xc0015c6ab0, 0xc00108b320, 0x4886160, 0xc000110718, 0xc0015c6ae0, 0xc0015c6ab0, 0xc00157df68) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x489e1a0, 0xc0015c6ab0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1809 +0xdd github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f96e0, 0xc00108b320, 0xc0007358f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2954 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453e70, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0010a0118, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0010a0100, 0xc002df6000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0010a0100, 0xc002df6000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00290e348, 0xc002df6000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0075e6b00, 0xc002df6000, 0x791, 0x791, 0xdb, 0x482c600, 0xc001618a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001f713d8, 0x482c340, 0xc0075e6b00, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001f71180, 0x48317e0, 0xc00290e348, 0x5, 0xc00290e348, 0xc006a4fc80) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001f71180, 0x40e5100, 0xc001f712b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001f71180, 0xc003536000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001924600, 0xc0013e2d58, 0x9, 0x9, 0xc00170efb8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001924600, 0xc0013e2d58, 0x9, 0x9, 0x9, 0xc007482b40, 0x485e320, 0xc005e5b420) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013e2d58, 0x9, 0x9, 0x482c140, 0xc001924600, 0x0, 0xc000000000, 0xc003f26000, 0x6a4fc00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013e2d20, 0xc006a4fc50, 0xc006a4fc50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000358e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 242218 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbe, 0x490bae0, 0xc006ee4780, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbe, 0xc002f806c0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc006c35500, 0x33, 0x0, 0x0, 0xc003ffbdd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1174 [semacquire]: sync.runtime_SemacquireMutex(0x726cb6c, 0xca51dca00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0x726cb68) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000001, 0x6be65da, 0x10, 0x82, 0xc00526d080, 0x145) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:857 +0x206 github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e1a0, 0xc006f07110, 0x1, 0x2, 0x3fc246f, 0x20, 0xc000dd4718, 0x3, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e1a0, 0xc006f07110, 0x1, 0x62843da100000001, 0x3fc246f, 0x20, 0xc000dd4718, 0x3, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Infof(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:95 github.com/cockroachdb/cockroach/pkg/sql.EventLogger.InsertEventRecord.func1(0x489e1a0, 0xc006f07110) /go/src/github.com/cockroachdb/cockroach/pkg/sql/event_log.go:130 +0x133 github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).commit(0xc00458b7a0, 0x489e1a0, 0xc006f07110, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:509 +0x1cb github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Commit(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:533 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).commitSQLTransaction(0xc0011b0000, 0x489e1a0, 0xc006f07110, 0x48c17a0, 0xc003d35490, 0x282bc4a, 0xc003d35730, 0xc0011b01d8, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:579 +0xa3 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).handleAutoCommit(0xc0011b0000, 0x489e1a0, 0xc006f07110, 0x48c17a0, 0xc003d35490, 0x3bac4e0, 0x736ebc, 0x0, 0x10) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1231 +0xbb github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState.func5(0xc0011b0000, 0xc000dd54b8, 0xc005dbd4a8, 0x489e1a0, 0xc006f07110, 0xc003d35730, 0xc005dbd498, 0x48606a0, 0x6fd682f, 0x48606a0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:211 +0xe3 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0011b0000, 0x489e1a0, 0xc006f07110, 0x48c17a0, 0xc003d35490, 0xc004faa7cb, 0xb6, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:447 +0xcc6 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0011b0000, 0x489e1a0, 0xc006f07110, 0x48c17a0, 0xc003d35490, 0xc004faa7cb, 0xb6, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:98 +0x4ec github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0011b0000, 0x489e0e0, 0xc003b63180, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1238 +0x1aa2 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0011b0000, 0x489e0e0, 0xc0015375c0, 0xc0004b2c40, 0x5400, 0x15000, 0xc0004b2cd8, 0xc000354eb0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00041d400, 0x489e0e0, 0xc0015375c0, 0xc0011b0000, 0x5400, 0x15000, 0xc0004b2cd8, 0xc000354eb0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444 +0xce github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0003ce78d, 0xc0003e87e0, 0x489e0e0, 0xc0015375c0, 0xc000354eb0, 0xc00041d400, 0xc0013fc000, 0x48c0ea0, 0xc0003e87c0, 0xc001126540, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584 +0x21c created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:513 +0x17b goroutine 1292 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec960) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f610, 0xc00108b320, 0xc00102f600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 8482 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0010e50e0, 0x48fa580, 0xc001a9c480, 0x485c120, 0xc0010e50e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc00432aa80, 0x0, 0xc002321620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc00432aa80, 0xc00191ae20, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc00432aa80, 0xc00191ae20, 0x40e1ba8, 0x489e1a0, 0xc001852db0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc003944900, 0xc0013d4330, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc003944900, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003b98d0, 0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc003944900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1494 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000fae900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1445 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001033dc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1360 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000403f40, 0xc000333b10, 0x5, 0x5, 0x32, 0xc006203868, 0xd1dafb) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000403f40, 0xc000333b10, 0x5, 0x5, 0xc11b10, 0xc0015d8620, 0xc006203888) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000ffd7d0, 0xc000333b10, 0x5, 0x5, 0x6f533a, 0xc0062038b0, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000ffd7d0, 0xc000333b10, 0x5, 0x5, 0x5, 0xc006203968, 0xf02137, 0x3a62440) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001aca300, 0xc000333b10, 0x5, 0x5, 0x7ee108, 0xc000ffd770, 0x3bbf7c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000333b00, 0x7fffffff, 0x71fd21, 0x40e43b8, 0xc006203a08, 0x20, 0xc001254c00, 0x40e43b8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000333b00, 0xc001aca300, 0x0, 0x0, 0x7fffffff, 0xc006e75bc0, 0x4886820, 0x72b56a8, 0x8, 0x6f8939, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000333b00, 0x7f086d36adf0, 0xc000284fd0, 0xc001aca300, 0x0, 0x0, 0x3dd1dc0, 0xc005b94dc0, 0x7fffffff, 0xc006e75bc0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001af60c0, 0x3dd1dc0, 0xc005b94dc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc000354170, 0x768c8d, 0xc9d048, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:588 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc000c9d000, 0x489e1a0, 0xc000ffd9e0, 0xc000110150, 0xc001032a40, 0xc006e75f50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:334 +0x341 github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1(0x489e1a0, 0xc000ffd9e0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:141 +0x9e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003541f0, 0xc000cfe090, 0xc001032ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1361 [semacquire]: sync.runtime_SemacquireMutex(0xc0003692b4, 0x7f086d499b00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc0003692b0) /usr/local/go/src/sync/mutex.go:134 +0x109 sync.(*Once).Do(0xc0003692b0, 0xc000d21f50) /usr/local/go/src/sync/once.go:40 +0x3b github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc000795e30) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354b10, 0xc00108b320, 0xc0002a1060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1446 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0016ad380, 0xc0010a4de0, 0xc000353340, 0xf, 0xc000e8f7a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000698ba0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355050, 0xc00108b320, 0xc000def270) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242815 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152b340, 0x489e120, 0xc00010c010, 0xba, 0x490bae0, 0xc006e350e0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c3cea0, 0x489e120, 0xc00010c010, 0xba, 0xc0028647e0, 0xc0025f1f38, 0xc006b1aae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc003c8b500, 0x33, 0x0, 0x0, 0xc003423b90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7333 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0010e50e0, 0x3, 0xc00290cd20, 0xc004195380, 0x48fa520, 0xc0028b1b80, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003423170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0028b1b00, 0xc00108b320, 0xc003423110) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6965 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453c00, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0010a1298, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0010a1280, 0xc001410000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0010a1280, 0xc001410000, 0x75b, 0x75b, 0xc003677938, 0x701, 0xc003ea8880) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcc4c8, 0xc001410000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0040f6780, 0xc001410000, 0x75b, 0x75b, 0xc00141000d, 0x9c, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc00637e780, 0xc001410000, 0x75b, 0x75b, 0xb9, 0x482c600, 0xc003677a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003ea8958, 0x482c340, 0xc00637e780, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003ea8700, 0x7f086d36ae18, 0xc0040f6780, 0x5, 0xc0040f6780, 0x9c) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003ea8700, 0x40e5100, 0xc003ea8838, 0x7f0866740260) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003ea8700, 0xc00278e000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003c28c60, 0xc0037321f8, 0x9, 0x9, 0xc0035f4408, 0xc0022c08c0, 0x93) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003c28c60, 0xc0037321f8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0037321f8, 0x9, 0x9, 0x482c140, 0xc003c28c60, 0x0, 0x300000000, 0xc00288bb00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0037321c0, 0xc004143b30, 0xc004143b30, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00419f500, 0xc003eaf1a0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0025ed980, 0x48fef60, 0xc00419f500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0025ed980, 0x48fef60, 0xc00419f500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1475 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0016ad380, 0xc000cc5ec0, 0xc000353340, 0xf, 0xc000b13680, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000c36d20) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003343e0, 0xc00108b320, 0xc00156e000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242615 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc0056220c0, 0x0, 0x1, 0x0, 0x3de2740) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0x6, 0xc003bccb40, 0xc0036cc000, 0x15c1014bd306c112) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0042d60c0, 0x33, 0x0, 0x0, 0xc004880990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1417 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0016f6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1447 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001120000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1748 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000cf1040, 0xc0022a6d80, 0xc000243dc0, 0xf, 0xc002480480, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000decc30) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00112e740, 0xc000cfe090, 0xc002456b90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1357 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0018d6000, 0xc0017705a0, 0x489e1a0, 0xc000ffd410) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1498 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc000cd24b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355630, 0xc00108b320, 0xc000314ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1285 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec810) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f530, 0xc00108b320, 0xc00102f500) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1813 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000358c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1749 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001120dc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2432 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc0002f0f20, 0x7f086677d630, 0xc00031cb80, 0x7f086677d630, 0xc00031cb80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc0002f0f20, 0x48fa760, 0xc00031cb80, 0xc02f00010301, 0xc00031cb80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc00037da20, 0x48eb940, 0xc002dd6180, 0x0, 0xc002321620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc00037da20, 0x48eb940, 0xc002dd6180, 0xc0002c4400, 0x40e1e60, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc00037da20, 0x48eb940, 0xc002dd6180, 0xc0002c4400, 0x40e1e60, 0x489e1a0, 0xc001108b40) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc001934300, 0xc0013d4b40, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc001934300, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003b98d0, 0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc001934300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2146 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc000327080, 0x7f086677d630, 0xc0018101a0, 0x7f086677d630, 0xc0018101a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc000327080, 0x48fa760, 0xc0018101a0, 0xc02f00010301, 0xc0018101a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0002ba420, 0x48eb940, 0xc002dd6000, 0x0, 0xc00133aae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0002ba420, 0x48eb940, 0xc002dd6000, 0xc002d62500, 0x40e1e60, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0002ba420, 0x48eb940, 0xc002dd6000, 0xc002d62500, 0x40e1e60, 0x489e1a0, 0xc0011bc570) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc000fae900, 0xc002dce100, 0xc000de86f0, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc000fae900, 0xc002dce100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002b2700, 0xc000ddc180, 0x48fef60, 0xc000fae900, 0xc002dce100) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1396 [select]: github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x489e1a0, 0xc0016b6570) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:44 +0x16b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c120, 0xc00108b320, 0xc0016b64b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1474 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001253a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2152 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00071e340, 0xc002d6efc0, 0xc002a43750, 0xf, 0xc0026c6ba0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0016c6810) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001810900, 0xc0011a2480, 0xc002e541e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1352 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001032240, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000ee6f60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000c04540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1350 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000c04540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1290 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec900) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f5d0, 0xc00108b320, 0xc00102f5c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1287 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec870) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f570, 0xc00108b320, 0xc00102f560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1291 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec930) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f5f0, 0xc00108b320, 0xc00102f5e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2372 [chan send]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:767 +0x1dc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8d0, 0xc0011a2480, 0xc0016be0a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2494 [IO wait]: internal/poll.runtime_pollWait(0x7f086d319f58, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001196b18, 0x72, 0x4900, 0x4978, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001196b00, 0xc004e66000, 0x4978, 0x4978, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001196b00, 0xc004e66000, 0x4978, 0x4978, 0xc002f40938, 0x4901, 0xc001710f80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001154230, 0xc004e66000, 0x4978, 0x4978, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00295f980, 0xc004e66000, 0x4978, 0x4978, 0xc004e6600d, 0x5c, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc007afb900, 0xc004e66000, 0x4978, 0x4978, 0x79, 0x482c600, 0xc002f40a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001711058, 0x482c340, 0xc007afb900, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001710e00, 0x7f086d36ae18, 0xc00295f980, 0x5, 0xc00295f980, 0x5c) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001710e00, 0x40e5100, 0xc001710f38, 0x5) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001710e00, 0xc0032b2000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002321680, 0xc0014050d8, 0x9, 0x9, 0xc0035f4908, 0xc0065e54a0, 0x53) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002321680, 0xc0014050d8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0014050d8, 0x9, 0x9, 0x482c140, 0xc002321680, 0x0, 0x500000000, 0xc003944900, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0014050a0, 0xc000c79f50, 0xc000c79f50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0024ca480, 0xc000cd3c80, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc001a74c00, 0x48fef60, 0xc0024ca480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc001a74c00, 0x48fef60, 0xc0024ca480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 243663 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc00626cde0, 0x0, 0x1, 0xc000ac1de0, 0x7ee93c) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0x6, 0xc005a485a0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc00027cbc0, 0x33, 0x0, 0x0, 0xc00673c7e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 242526 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152b340, 0x489e120, 0xc00010c010, 0xba, 0x490bae0, 0xc0060d1ec0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c3cea0, 0x489e120, 0xc00010c010, 0xba, 0xc005c86c60, 0xc0024ae800, 0x15c1014bd3e20042) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc003c8afc0, 0x33, 0x0, 0x0, 0xc0017d28a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1510 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4020) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335490, 0xc00108b320, 0xc000335480) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 241897 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xb9, 0x490bae0, 0xc0027a0240, 0x0, 0x1, 0xc0014086c0, 0xc0016cde28) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xb9, 0xc006eddc20, 0xc0042a5800, 0x15c1014bce629aab) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc003cb3800, 0x33, 0x0, 0x0, 0xc005fe9f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1376 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001772700, 0xc001aee000, 0x489e1a0, 0xc0011bc1e0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1359 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc000c9d000, 0x48f9f20, 0xc000354170, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:193 +0x625 github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x3e992e0, 0xc000c9d000, 0x48eb940, 0xc001af60c0, 0x0, 0xc000ee6d20) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:569 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3e992e0, 0xc000c9d000, 0x48eb940, 0xc001af60c0, 0xc000333b20, 0x40df160, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3e992e0, 0xc000c9d000, 0x48eb940, 0xc001af60c0, 0xc000333b20, 0x40df160, 0x489e1a0, 0xc000ffd830) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc001486a80, 0xc001aca300, 0xc000aa6ab0, 0x6f9bba0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc001486a80, 0xc001aca300, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000352e70, 0xc000ddc180, 0x48fef60, 0xc001486a80, 0xc001aca300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1509 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4003) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335470, 0xc00108b320, 0xc000335450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1358 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000403ae0, 0xc0003338b0, 0x5, 0x5, 0x59, 0xc007bd72c0, 0xa) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000403ae0, 0xc0003338b0, 0x5, 0x5, 0xc11b10, 0xc0015d85a0, 0xc0048bb970) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000ffd560, 0xc0003338b0, 0x5, 0x5, 0x0, 0xc0048bb998, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000ffd560, 0xc0003338b0, 0x5, 0x5, 0x5, 0x14dfbb6, 0xc007bd7201, 0xc0048bba50) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001aca200, 0xc0003338b0, 0x5, 0x5, 0xc0029a8180, 0x0, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003338a0, 0x7fffffff, 0xc0075e6c00, 0x3, 0x4, 0x3, 0x4, 0x9) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003338a0, 0xc001aca200, 0x0, 0x0, 0x7fffffff, 0xc0048bbcb0, 0x4886820, 0x72b56a8, 0x0, 0xa, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003338a0, 0x7f086d36adf0, 0xc000284fd0, 0xc001aca200, 0x0, 0x0, 0x3dd1ee0, 0xc0076d0e60, 0x7fffffff, 0xc0048bbcb0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001502280, 0x3dd1ee0, 0xc0076d0e60, 0xc0048bbcb0, 0xc0048bbd30, 0xc0076d0e60) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001502280, 0xc0048bbd40, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0017705a0, 0xc0048bbdf8, 0xc0048bbde8, 0x0, 0x48) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0017705a0, 0x3dd1ee0, 0xc0076d0e60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc00023aad0, 0x489e1a0, 0xc000ffd620, 0xc001a74d80) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:553 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0xc000e8f260, 0x48f9ec0, 0xc00023aad0, 0xc000be0820, 0x489e1a0, 0xc000ffd620, 0xc001a74d80, 0x30, 0x3d2f600) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:322 +0x4a github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x489e1a0, 0xc000ffd620) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:337 +0xde github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00023aff0, 0xc00108b320, 0xc001032840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1005 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001d1c000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1354 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001486a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1353 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001032200, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000ee7500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001486a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1355 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44cb98, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001502098, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001502080, 0xc00188d000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001502080, 0xc00188d000, 0x75b, 0x75b, 0xc001226938, 0x701, 0xc001a70500) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110068, 0xc00188d000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc000ee6840, 0xc00188d000, 0x75b, 0x75b, 0xc00188d00d, 0x19d, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0058dfc20, 0xc00188d000, 0x75b, 0x75b, 0x1ba, 0x482c600, 0xc001226a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001a705d8, 0x482c340, 0xc0058dfc20, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001a70380, 0x7f086d36ae18, 0xc000ee6840, 0x5, 0xc000ee6840, 0x19d) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001a70380, 0x40e5100, 0xc001a704b8, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001a70380, 0xc001010000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000ee6d80, 0xc00028c2d8, 0x9, 0x9, 0xc001032988, 0xc006849ba0, 0x194) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000ee6d80, 0xc00028c2d8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00028c2d8, 0x9, 0x9, 0x482c140, 0xc000ee6d80, 0x0, 0x300000000, 0xc001aca300, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00028c2a0, 0xc000ffc720, 0xc000ffc720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001486a80, 0xc000ffcbd0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc001486a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc001486a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1512 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x401d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003354e0, 0xc00108b320, 0xc0003354d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1513 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x400d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335500, 0xc00108b320, 0xc0003354f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1507 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x405f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335420, 0xc00108b320, 0xc000335410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2431 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001430140, 0xc001760fb0, 0x5, 0x5, 0x0, 0xc006e620e0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001430140, 0xc001760fb0, 0x5, 0x5, 0xc11b10, 0xc00026fd80, 0xc002c3d858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001114870, 0xc001760fb0, 0x5, 0x5, 0xc001f64d00, 0xc002c3d880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001114870, 0xc001760fb0, 0x5, 0x5, 0x5, 0xc0012a7380, 0xc002c3d948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003786400, 0xc001760fb0, 0x5, 0x5, 0xc004cc0000, 0x40, 0xc000d66700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001760fa0, 0x7fffffff, 0xc0e5c1, 0xc0029e8e40, 0x58, 0x58, 0xc007944060, 0xc000423e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001760fa0, 0xc003786400, 0x0, 0x0, 0x7fffffff, 0xc002c3db98, 0x4886820, 0x72b56a8, 0x40, 0xc003040d3f, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001760fa0, 0x7f086d36adf0, 0xc000284fd0, 0xc003786400, 0x0, 0x0, 0x3de9580, 0xc00759f260, 0x7fffffff, 0xc002c3db98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0010a0280, 0x3de9580, 0xc00759f260, 0xc002c3db98, 0x0, 0xc00759f260) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0010a0280, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001434c60, 0xc002c3dce0, 0xc002c3dcd0, 0xc00299c000, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001434c60, 0x3de9580, 0xc00759f260, 0xc001f64cc0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0011592b0, 0xc001f64cc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc001114780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c930, 0xc0011a2480, 0xc0002c40e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2966 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002a6d2c0, 0xc0003278b0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002a6d2c0, 0xc0003278b0, 0x5, 0x5, 0xc11b10, 0xc0003e62c0, 0xc004ef7978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00190c570, 0xc0003278b0, 0x5, 0x5, 0x74cff0, 0xc004ef79a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00190c570, 0xc0003278b0, 0x5, 0x5, 0x5, 0xf, 0xc000636900, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00323c800, 0xc0003278b0, 0x5, 0x5, 0x83e500, 0xc0002f5170, 0xc004ef7a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003278a0, 0x7fffffff, 0x3c06120, 0xc000636900, 0xc004ef7a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003278a0, 0xc00323c800, 0x0, 0x0, 0x7fffffff, 0xc004ef7cb8, 0x4886820, 0x72b56a8, 0x0, 0xc0024ef4c0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003278a0, 0x7f086d36adf0, 0xc000284fd0, 0xc00323c800, 0x0, 0x0, 0x3de8800, 0xc003b63400, 0x7fffffff, 0xc004ef7cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc002dc4b00, 0x3de8800, 0xc003b63400, 0xc004ef7cb8, 0xc0049c2000, 0xc003b63400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc002dc4b00, 0xc000092f70, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0016f4b40, 0xc004ef7e00, 0xc004ef7df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0016f4b40, 0x3de8800, 0xc003b63400, 0x2c5e8e0, 0xc000ce8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc00198cec0, 0x489e1a0, 0xc00190c600, 0xc004a53700) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc00198cec0, 0xc0004a5ac0, 0xc000214870, 0x489e1a0, 0xc00190c600, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc00190c600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00198cee0, 0xc000cfe090, 0xc00190c5a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1400 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44ce08, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001503f98, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001503f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001503f80, 0xc0025e4701, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110710, 0xc001565e80, 0xc001565e88, 0x28) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110710, 0x40e2690, 0xc001414200, 0x48faa00, 0xc0025e4760) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc001414200, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1() /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1731 +0x2f sync.(*Once).Do(0xc0003692b0, 0xc000d8f750) /usr/local/go/src/sync/once.go:44 +0xb3 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc0016b6540) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c230, 0xc00108b320, 0xc001760460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 8443 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214960, 0x2, 0xc002320000, 0xc004195400, 0x48fa520, 0xc0016b4310, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc0010f54a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b41c0, 0xc0011a2480, 0xc0010f5440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1508 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4005) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335440, 0xc00108b320, 0xc000335430) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1289 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec7b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f5b0, 0xc00108b320, 0xc00102f5a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1511 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fff) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003354b0, 0xc00108b320, 0xc0003354a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1514 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4009) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335520, 0xc00108b320, 0xc000335510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2147 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000ee4e10, 0xa2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000ee4e00) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc00083e5b0, 0x489e1a0, 0xc0011bc570, 0xc002d6e3c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc0011bc570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e1a0, 0xc0011bc570, 0xc0015d80e0, 0x1d, 0x0, 0x0, 0xc002d62540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1146 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x489e1a0, 0xc001a56f90) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:171 +0x536 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f9680, 0xc00108b320, 0xc0003e8ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1487 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00152b340, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc000321680, 0x22, 0x0, 0x0, 0xc0003353a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1397 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x489e1a0, 0xc0016b65a0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c1e0, 0xc00108b320, 0xc001760440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6870 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453a60, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc003c12018, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc003c12000, 0xc0015da000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc003c12000, 0xc0015da000, 0x75b, 0x75b, 0xc000815938, 0x701, 0x294a5294a5) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc004b940b8, 0xc0015da000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00408ea80, 0xc0015da000, 0x75b, 0x75b, 0x8, 0x107663801f22, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0021cfea0, 0xc0015da000, 0x75b, 0x75b, 0x203000, 0x140, 0xc000815a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00151e958, 0x482c340, 0xc0021cfea0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00151e700, 0x7f086d36ae18, 0xc00408ea80, 0x5, 0xc00408ea80, 0x0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00151e700, 0x40e5100, 0xc00151e838, 0xc006866960) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00151e700, 0xc002980000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc004291b00, 0xc00115a3b8, 0x9, 0x9, 0xc0039b10c8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc004291b00, 0xc00115a3b8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00115a3b8, 0x9, 0x9, 0x482c140, 0xc004291b00, 0x0, 0xc900000000, 0xc006d7ff00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00115a380, 0xc003f256b0, 0xc003f256b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003981500, 0xc003f9b8c0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc001a74c00, 0x48fef60, 0xc003981500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc001a74c00, 0x48fef60, 0xc003981500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1304 [select]: net.(*pipe).read(0xc00140e280, 0xc0015f8000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc00140e280, 0xc0015f8000, 0x791, 0x791, 0xc0015f800d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc007557ce0, 0xc0015f8000, 0x791, 0x791, 0x2e, 0x482c600, 0xc001780ac0) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001d22258, 0x482c340, 0xc007557ce0, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001d22000, 0x7f086d451a50, 0xc00140e280, 0x5, 0xc00140e280, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001d22000, 0x40e5100, 0xc001d22138, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001d22000, 0xc000f28000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001466960, 0xc0003d2ab8, 0x9, 0x9, 0x6fe9a9, 0xc00156d500, 0xc001780e40) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001466960, 0xc0003d2ab8, 0x9, 0x9, 0x9, 0x0, 0xc000637498, 0x74a6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003d2ab8, 0x9, 0x9, 0x482c140, 0xc001466960, 0x0, 0xc000000000, 0xc001780ea7, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003d2a80, 0xc006aefc60, 0xc006aefc60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000358a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1489 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x489e1a0, 0xc000683c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:151 +0x4a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003353e0, 0xc00108b320, 0xc0002f1f00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1377 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc0011bc390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018100f0, 0xc00108b320, 0xc002d62340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1264 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc0016b7f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f80f0, 0xc00108b320, 0xc0003bfda0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1419 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001252580, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000cc4060, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0016f6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1404 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00142e2c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1378 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44ced8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001503f18, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001503f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001503f00, 0xc000b01500, 0x7f086d49af80, 0xc00171d800) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc0001106f8, 0x71f47f, 0xc000ee4830, 0xc000d64ef0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc0001106f8, 0x40e2688, 0xc0014141c0, 0xc0003b84a0, 0x4906de0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc0014141c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x489e1a0, 0xc000b59020) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1764 +0x2f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8140, 0xc00108b320, 0xc0003bfe40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1401 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc000be0820, 0x489e1a0, 0xc0016b66c0, 0xc001a74d80, 0x48f9ec0, 0xc00023aad0, 0xc00108b320, 0xc0016d8454, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:366 +0x344 github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x489e1a0, 0xc0016b66c0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:125 +0x3c2 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c2e0, 0xc00108b320, 0xc0010ec5c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1265 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x489e1a0, 0xc000b58ff0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1751 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8130, 0xc00108b320, 0xc0003bfde0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1284 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec7e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f4f0, 0xc00108b320, 0xc00102f4e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 657 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x489e1a0, 0xc001030270) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:478 +0x52 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e480, 0xc00108b320, 0xc00037d240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1418 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44cac8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001502a18, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001502a00, 0xc0010f2800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001502a00, 0xc0010f2800, 0x791, 0x791, 0x203000, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110470, 0xc0010f2800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc003c27320, 0xc0010f2800, 0x791, 0x791, 0x73, 0x482c600, 0xc000817a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00176bad8, 0x482c340, 0xc003c27320, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00176b880, 0x48317e0, 0xc000110470, 0x5, 0xc000110470, 0x56) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00176b880, 0x40e5100, 0xc00176b9b8, 0xc000817d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00176b880, 0xc001186000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000c2a720, 0xc000367a78, 0x9, 0x9, 0xc1c86c, 0xc0016f6168, 0xc000b90900) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000c2a720, 0xc000367a78, 0x9, 0x9, 0x9, 0x4d, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000367a78, 0x9, 0x9, 0x482c140, 0xc000c2a720, 0x0, 0x4d00000000, 0x10000000b, 0xb0000004d) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000367a40, 0xc000b90870, 0xc000b90870, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0016f6000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1476 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc000839260, 0x489e1a0, 0xc000cd2480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:134 +0x333 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000334bd0, 0xc00108b320, 0xc000334bc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1504 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001415750, 0x1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001415740) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc000143c20, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:437 +0x89 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc002418000, 0x489e0e0, 0xc001415780, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1191 +0x52 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc002418000, 0x489e0e0, 0xc001415780, 0xc001660100, 0x5400, 0x15000, 0xc001660198, 0xc00112e140, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc001324500, 0x489e0e0, 0xc001415780, 0xc002418000, 0x5400, 0x15000, 0xc001660198, 0xc00112e140, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444 +0xce github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0002b386d, 0xc0002e6820, 0x489e0e0, 0xc001415780, 0xc00112e140, 0xc001324500, 0xc000143600, 0x48c0ea0, 0xc0002e6800, 0xc0022a6360, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584 +0x21c created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:513 +0x17b goroutine 1403 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0016ad380, 0xc00192cc60, 0xc0003e42f0, 0xf, 0xc001164420, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0016b6840) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c580, 0xc00108b320, 0xc000bca9b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2077 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002c32000, 0xc000c8a000, 0x489e1a0, 0xc0015c6600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 651 [select]: database/sql.(*DB).connectionResetter(0xc0011b8180, 0x489e0e0, 0xc000ee5940) /usr/local/go/src/database/sql/sql.go:1013 +0xfb created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:671 +0x194 goroutine 2430 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001430320, 0xc0017611b0, 0x5, 0x5, 0xba, 0xc00191e960, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001430320, 0xc0017611b0, 0x5, 0x5, 0xc11b10, 0xc00026fe20, 0xc002f17858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001114c90, 0xc0017611b0, 0x5, 0x5, 0xc0065fd740, 0xc002f17880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001114c90, 0xc0017611b0, 0x5, 0x5, 0x5, 0xc0012a7680, 0xc002f17948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003786500, 0xc0017611b0, 0x5, 0x5, 0x40, 0x40, 0xc004364000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0017611a0, 0x7fffffff, 0xc0e5c1, 0xc002c60420, 0x58, 0x58, 0xc005211140, 0xc00198c4b0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0017611a0, 0xc003786500, 0x0, 0x0, 0x7fffffff, 0xc002f17b98, 0x4886820, 0x72b56a8, 0x0, 0xc0063a6b6b, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0017611a0, 0x7f086d36adf0, 0xc000284fd0, 0xc003786500, 0x0, 0x0, 0x3de9580, 0xc006ee3080, 0x7fffffff, 0xc002f17b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0010a0300, 0x3de9580, 0xc006ee3080, 0xc002f17b98, 0x0, 0xc006ee3080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0010a0300, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001434ea0, 0xc002f17ce0, 0xc002f17cd0, 0x72b56a8, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001434ea0, 0x3de9580, 0xc006ee3080, 0xc0065fd700, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc001159340, 0xc0065fd700, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc001114bd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8c0, 0xc0011a2480, 0xc0002c40a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2081 [chan send]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:767 +0x1dc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c870, 0xc0011a2480, 0xc0016be040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 240051 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbd, 0x490bae0, 0xc0027b7ce0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbd, 0xc006699680, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc004861700, 0x33, 0x0, 0x0, 0xc006ea44e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 656 [select]: github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc00108b320, 0xc000882790) /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x14a created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49 goroutine 2370 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c8a0, 0xc0011a2480, 0xc0016be060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1399 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x489e1a0, 0xc000ffc0f0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1400 +0x33c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106c220, 0xc00108b320, 0xc00106c210) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2181 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00152a2a0, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0016f8240, 0x22, 0x0, 0x0, 0xc00102e390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1263 [select]: github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x40e5438, 0xc000403360, 0xc0007b2b60, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11b created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x105 goroutine 1288 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000dec8a0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f590, 0xc00108b320, 0xc00102f580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2377 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc000c76340, 0x489e1a0, 0xc0019698f0, 0xc0011a2480, 0x27b3037b, 0xed500a8d2, 0x0, 0xc0013e5800, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:217 +0x1d1 github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc0013e5800, 0xc0016c6d01) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:279 +0x81 github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc0016c6de0, 0xc007633100) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x18d github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x489e1a0, 0xc0019698f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:277 +0x19b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c970, 0xc0011a2480, 0xc0016be160) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 939 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44cfa8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001ac6218, 0x72, 0x700, 0x719, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001ac6200, 0xc0015f9000, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001ac6200, 0xc0015f9000, 0x719, 0x719, 0x203001, 0x7f086d49a1f0, 0x0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcc018, 0xc0015f9000, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc005bd98c0, 0xc0015f9000, 0x719, 0x719, 0x2, 0x1, 0xc001751618) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00180f058, 0x482c340, 0xc005bd98c0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00180ee00, 0x48317e0, 0xc000bcc018, 0x5, 0xc000bcc018, 0xc0045d1c00) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00180ee00, 0x40e5100, 0xc00180ef38, 0xed500a8d3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00180ee00, 0xc001124000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc0003e8780, 0xc001124000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1896 +0x110 bufio.(*Reader).fill(0xc0013fc048) /usr/local/go/src/bufio/bufio.go:100 +0x10f bufio.(*Reader).ReadByte(0xc0013fc048, 0xc0013fc620, 0xed500a8d2, 0x0) /usr/local/go/src/bufio/bufio.go:249 +0x39 github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc0013fc7a0, 0x486e520, 0xc0013fc048, 0x4860120, 0x72b56a8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc0013fc000, 0x489e0e0, 0xc0011e4000, 0xc000354e90, 0xc00041d400, 0x5400, 0x15000, 0xc0004b2cd8, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:313 +0x474 github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x489e0e0, 0xc0011e4000, 0x48f9e60, 0xc00180ee00, 0xc000ea504a, 0x4, 0xc001250ff0, 0x485eca0, 0xc000b580c0, 0x4000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:168 +0x18d github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc0004b2b40, 0x489e0e0, 0xc0011e4000, 0x48f9e60, 0xc00180ee00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:536 +0x87e github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x48faa00, 0xc000bcc018) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1813 +0x165 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc000cfe090, 0x489e1a0, 0xc000dec210, 0xc000c74b20, 0x48faa00, 0xc000bcc018, 0xc000dec240) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xa9 created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b goroutine 2382 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0010de640, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00295fce0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0018d2c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1420 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001252540, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001126720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00149db00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1387 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x489e1a0, 0xc00142c180) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1228 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8410, 0xc00108b320, 0xc00184a260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1386 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001414500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1351 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44cc68, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0015a6298, 0x72, 0x3100, 0x3158, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0015a6280, 0xc002749500, 0x3158, 0x3158, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0015a6280, 0xc002749500, 0x3158, 0x3158, 0x203001, 0x0, 0x314b) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcc118, 0xc002749500, 0x3158, 0x3158, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0075e6b80, 0xc002749500, 0x3158, 0x3158, 0xc0, 0x482c600, 0xc001567a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0018d65d8, 0x482c340, 0xc0075e6b80, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0018d6380, 0x48317e0, 0xc000bcc118, 0x5, 0xc000bcc118, 0xa3) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0018d6380, 0x40e5100, 0xc0018d64b8, 0xc001567d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0018d6380, 0xc001082000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000ee6ea0, 0xc00028c3b8, 0x9, 0x9, 0xc1c86c, 0xc000c046a8, 0xc000ffc900) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000ee6ea0, 0xc00028c3b8, 0x9, 0x9, 0x9, 0x9a, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00028c3b8, 0x9, 0x9, 0x482c140, 0xc000ee6ea0, 0x0, 0x9a00000000, 0x100000003, 0x30000009a) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00028c380, 0xc000ffc840, 0xc000ffc840, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000c04540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1384 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x489e1a0, 0xc000ded200) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1198 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f82d0, 0xc00108b320, 0xc0013fa090) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1383 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc001a74c00, 0x4886ea0, 0xc0003bff00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x1e9 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x489e1a0, 0xc000ded170) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x46 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f81e0, 0xc00108b320, 0xc0003bff20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1382 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x489e1a0, 0xc000ded1d0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1706 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f81d0, 0xc00108b320, 0xc001414240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2964 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214870, 0x3, 0xc00228aba0, 0xc0004a5ac0, 0x48fa520, 0xc00198cec0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc00190c390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00198ce50, 0xc000cfe090, 0xc00190c300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2497 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0017ca000, 0xc0011c1560, 0x489e1a0, 0xc000791aa0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1747 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ac6e40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1381 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x489e1a0, 0xc000ded1a0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1654 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f81b0, 0xc00108b320, 0xc0003bfee0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1380 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 crypto/tls.(*listener).Accept(0xc0003bfe60, 0x3d2f600, 0xc000b58fc0, 0x3a88520, 0x6f95560) /usr/local/go/src/crypto/tls/tls.go:62 +0x37 net/http.(*Server).Serve(0xc0016340d0, 0x486e6e0, 0xc0003bfe60, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x489e1a0, 0xc000b58d80) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1791 +0x3f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8160, 0xc00108b320, 0xc0003bfe80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1402 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0010eca80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2779 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002ed89c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002a6ea20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0010281c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2078 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002457db0, 0xc002c686b0, 0x5, 0x5, 0x7260160, 0x32, 0x15c1014be6fb46a4) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002457db0, 0xc002c686b0, 0x5, 0x5, 0xc11b10, 0xc002a4f000, 0xc0023fb970) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0015c6780, 0xc002c686b0, 0x5, 0x5, 0xc00004e000, 0xc0023fb998, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0015c6780, 0xc002c686b0, 0x5, 0x5, 0x5, 0xc0023fba48, 0xc372ac, 0xc0071894a0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0015a9400, 0xc002c686b0, 0x5, 0x5, 0x83e57d, 0xc0071894a0, 0x4831580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc002c686a0, 0x7fffffff, 0x8, 0xc006b4a720, 0xc008170860, 0x600, 0x3dd1ee0, 0xc0071894a0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc002c686a0, 0xc0015a9400, 0x0, 0x0, 0x7fffffff, 0xc0023fbcb0, 0x4886820, 0x72b56a8, 0x0, 0x30006a255fb79e3, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc002c686a0, 0x7f086d36adf0, 0xc000284fd0, 0xc0015a9400, 0x0, 0x0, 0x3dd1ee0, 0xc005f485f0, 0x7fffffff, 0xc0023fbcb0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000eaf680, 0x3dd1ee0, 0xc005f485f0, 0xc0023fbcb0, 0xc0023fbd30, 0xc005f485f0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000eaf680, 0xc0023fbd40, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000c8a000, 0xc0023fbdf8, 0xc0023fbde8, 0x0, 0x48) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000c8a000, 0x3dd1ee0, 0xc005f485f0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc0003031c0, 0x489e1a0, 0xc0015c6810, 0xc0025edb00) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:553 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0xc0012caf60, 0x48f9ec0, 0xc0003031c0, 0xc000ddfc70, 0x489e1a0, 0xc0015c6810, 0xc0025edb00, 0x30, 0x3d2f600) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:322 +0x4a github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x489e1a0, 0xc0015c6810) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:337 +0xde github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000303360, 0xc0011a2480, 0xc002cb80c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1666 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c79050) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da20, 0xc00108b320, 0xc000301c40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2182 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00152a380, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0016f82a0, 0x22, 0x0, 0x0, 0xc00102e3a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1472 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c78ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da00, 0xc00108b320, 0xc000301c00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1473 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c79020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da10, 0xc00108b320, 0xc000301c20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1961 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0013c96b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003545b0, 0xc0011a2480, 0xc000354590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1667 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c79080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da30, 0xc00108b320, 0xc000301c60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1471 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0010ed1c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00155a060, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0011661c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1470 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44c928, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0015a6418, 0x72, 0x3200, 0x3218, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0015a6400, 0xc002746000, 0x3218, 0x3218, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0015a6400, 0xc002746000, 0x3218, 0x3218, 0x203001, 0x0, 0x320b) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcc130, 0xc002746000, 0x3218, 0x3218, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc006de3040, 0xc002746000, 0x3218, 0x3218, 0x59, 0x482c600, 0xc000819a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0018d6958, 0x482c340, 0xc006de3040, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0018d6700, 0x48317e0, 0xc000bcc130, 0x5, 0xc000bcc130, 0x3c) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0018d6700, 0x40e5100, 0xc0018d6838, 0xc000819d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0018d6700, 0xc001230000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001127a40, 0xc0013e21f8, 0x9, 0x9, 0xc1c86c, 0xc001166328, 0xc0006838f0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001127a40, 0xc0013e21f8, 0x9, 0x9, 0x9, 0x33, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013e21f8, 0x9, 0x9, 0x482c140, 0xc001127a40, 0x0, 0x3300000000, 0x10000004b, 0x4b00000033) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013e21c0, 0xc0006837d0, 0xc0006837d0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0011661c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1469 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0011661c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1466 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00142e580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1668 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c790b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da40, 0xc00108b320, 0xc000301c80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1493 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001253d00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00155a0c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000fae900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1515 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335540, 0xc00108b320, 0xc000335530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1516 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4019) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c020, 0xc00108b320, 0xc000335550) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1517 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4001) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c050, 0xc00108b320, 0xc00038c040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1518 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4008) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c080, 0xc00108b320, 0xc00038c070) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1519 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x400b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c0b0, 0xc00108b320, 0xc00038c090) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1520 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x400a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c160, 0xc00108b320, 0xc00038c0e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1521 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x401f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa61b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa61b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c190, 0xc00108b320, 0xc00038c170) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1522 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4017) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa61e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa61e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c490, 0xc00108b320, 0xc00038c1a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1523 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4026) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c4e0, 0xc00108b320, 0xc00038c4d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1524 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4013) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c540, 0xc00108b320, 0xc00038c530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1525 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x400c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c580, 0xc00108b320, 0xc00038c560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1526 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4012) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa62a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa62a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c5a0, 0xc00108b320, 0xc00038c590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1527 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4025) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa62d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa62d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c5c0, 0xc00108b320, 0xc00038c5b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1528 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x402a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c5e0, 0xc00108b320, 0xc00038c5d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1529 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4034) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c610, 0xc00108b320, 0xc00038c600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1530 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4011) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c630, 0xc00108b320, 0xc00038c620) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1531 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x400e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa69c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa69c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c660, 0xc00108b320, 0xc00038c640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1532 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x401a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6a20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6a20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c680, 0xc00108b320, 0xc00038c670) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1533 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4015) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa6ae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa6ae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c6a0, 0xc00108b320, 0xc00038c690) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1534 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4018) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c6d0, 0xc00108b320, 0xc00038c6c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1535 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4014) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c700, 0xc00108b320, 0xc00038c6e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1536 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4036) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa73b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa73b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c730, 0xc00108b320, 0xc00038c710) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1537 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x401c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c760, 0xc00108b320, 0xc00038c740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1538 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4021) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c780, 0xc00108b320, 0xc00038c770) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1539 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4037) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c7d0, 0xc00108b320, 0xc00038c7b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1540 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x401b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c850, 0xc00108b320, 0xc00038c7e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1541 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4024) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c880, 0xc00108b320, 0xc00038c860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1542 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4016) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa76b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa76b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c8a0, 0xc00108b320, 0xc00038c890) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1543 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4004) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa76e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa76e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c8d0, 0xc00108b320, 0xc00038c8c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1544 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4022) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c920, 0xc00108b320, 0xc00038c910) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1545 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x402e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c940, 0xc00108b320, 0xc00038c930) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1546 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4035) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c960, 0xc00108b320, 0xc00038c950) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1547 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4023) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa77a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa77a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c980, 0xc00108b320, 0xc00038c970) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1548 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4033) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c9b0, 0xc00108b320, 0xc00038c9a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1549 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x401e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa78c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa78c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c9d0, 0xc00108b320, 0xc00038c9c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1550 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4010) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa78f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa78f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c9f0, 0xc00108b320, 0xc00038c9e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1551 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4027) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ca10, 0xc00108b320, 0xc00038ca00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1552 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4044) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ca30, 0xc00108b320, 0xc00038ca20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1553 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x402d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ca50, 0xc00108b320, 0xc00038ca40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1554 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x402f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa79b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa79b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ca80, 0xc00108b320, 0xc00038ca70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1555 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x403b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa79e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa79e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038caa0, 0xc00108b320, 0xc00038ca90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1556 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x404d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cac0, 0xc00108b320, 0xc00038cab0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1557 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4028) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038caf0, 0xc00108b320, 0xc00038cae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1558 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x402c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cb20, 0xc00108b320, 0xc00038cb00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1559 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4042) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cb60, 0xc00108b320, 0xc00038cb50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1560 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x403a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cb90, 0xc00108b320, 0xc00038cb80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1561 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x404b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cbc0, 0xc00108b320, 0xc00038cba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1562 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4032) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cbf0, 0xc00108b320, 0xc00038cbd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1563 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4049) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cc60, 0xc00108b320, 0xc00038cc00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1564 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x402b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cc80, 0xc00108b320, 0xc00038cc70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1565 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4041) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cca0, 0xc00108b320, 0xc00038cc90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1566 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4029) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ccc0, 0xc00108b320, 0xc00038ccb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1567 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4031) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cce0, 0xc00108b320, 0xc00038ccd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1568 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4043) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cd00, 0xc00108b320, 0xc00038ccf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1569 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4038) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cd20, 0xc00108b320, 0xc00038cd10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1570 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4030) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cd40, 0xc00108b320, 0xc00038cd30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1571 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4040) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cd60, 0xc00108b320, 0xc00038cd50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1572 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x403d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cd80, 0xc00108b320, 0xc00038cd70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1573 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x403f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cda0, 0xc00108b320, 0xc00038cd90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1574 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4007) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000683c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000683c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ce00, 0xc00108b320, 0xc00038cdf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1575 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x403c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ce20, 0xc00108b320, 0xc00038ce10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1576 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x403e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ce50, 0xc00108b320, 0xc00038ce40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1577 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4039) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ce80, 0xc00108b320, 0xc00038ce60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1578 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4045) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7e60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7e60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ceb0, 0xc00108b320, 0xc00038ce90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1579 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4064) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038ced0, 0xc00108b320, 0xc00038cec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1580 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fe6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cf10, 0xc00108b320, 0xc00038cf00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1581 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4046) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7ef0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7ef0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cf80, 0xc00108b320, 0xc00038cf70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1582 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4061) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cfb0, 0xc00108b320, 0xc00038cfa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1583 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4047) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038cfe0, 0xc00108b320, 0xc00038cfd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1584 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4053) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d010, 0xc00108b320, 0xc00038d000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1585 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x404a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d050, 0xc00108b320, 0xc00038d020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1586 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4054) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d080, 0xc00108b320, 0xc00038d070) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1587 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4051) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c780c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c780c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d0b0, 0xc00108b320, 0xc00038d0a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1588 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x404e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d0e0, 0xc00108b320, 0xc00038d0d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1589 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x405b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d150, 0xc00108b320, 0xc00038d120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1590 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4050) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c781b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c781b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d170, 0xc00108b320, 0xc00038d160) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1591 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x404c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c781e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c781e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d190, 0xc00108b320, 0xc00038d180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1592 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4056) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c782a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c782a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d1b0, 0xc00108b320, 0xc00038d1a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1593 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4055) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c782d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c782d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d1d0, 0xc00108b320, 0xc00038d1c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1594 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4057) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d1f0, 0xc00108b320, 0xc00038d1e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1595 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x405c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d210, 0xc00108b320, 0xc00038d200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1596 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4059) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c783c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c783c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d230, 0xc00108b320, 0xc00038d220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1597 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fec) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c783f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c783f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d250, 0xc00108b320, 0xc00038d240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1598 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4052) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d270, 0xc00108b320, 0xc00038d260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1599 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fed) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d290, 0xc00108b320, 0xc00038d280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1600 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4062) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d2b0, 0xc00108b320, 0xc00038d2a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1601 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x404f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c784b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c784b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d2d0, 0xc00108b320, 0xc00038d2c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1602 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fe7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d2f0, 0xc00108b320, 0xc00038d2e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1603 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fe9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d310, 0xc00108b320, 0xc00038d300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1604 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x405d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c785a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c785a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d330, 0xc00108b320, 0xc00038d320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1605 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4060) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c785d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c785d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d350, 0xc00108b320, 0xc00038d340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1606 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fef) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d370, 0xc00108b320, 0xc00038d360) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1607 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x405a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d390, 0xc00108b320, 0xc00038d380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1608 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3feb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d3b0, 0xc00108b320, 0xc00038d3a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1609 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ffb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c786c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c786c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d3d0, 0xc00108b320, 0xc00038d3c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1610 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4058) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c786f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c786f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d400, 0xc00108b320, 0xc00038d3e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1611 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4048) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000aa7da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000aa7da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d430, 0xc00108b320, 0xc00038d410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1612 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4000) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d460, 0xc00108b320, 0xc00038d440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1613 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x405e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c787b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c787b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d490, 0xc00108b320, 0xc00038d470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1614 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c787e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c787e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d4c0, 0xc00108b320, 0xc00038d4a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1615 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d540, 0xc00108b320, 0xc00038d4d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1616 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d570, 0xc00108b320, 0xc00038d560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1617 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d5e0, 0xc00108b320, 0xc00038d590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1618 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4006) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c788d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c788d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d620, 0xc00108b320, 0xc00038d610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1619 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d640, 0xc00108b320, 0xc00038d630) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1620 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4063) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d660, 0xc00108b320, 0xc00038d650) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1621 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ffa) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c789f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c789f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d680, 0xc00108b320, 0xc00038d670) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1622 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ffd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78a20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78a20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d6a0, 0xc00108b320, 0xc00038d690) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1623 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fee) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78a50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78a50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d6c0, 0xc00108b320, 0xc00038d6b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1624 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4065) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d6e0, 0xc00108b320, 0xc00038d6d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1625 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ffc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78ab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78ab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d700, 0xc00108b320, 0xc00038d6f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1626 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78ae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78ae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d720, 0xc00108b320, 0xc00038d710) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1627 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fe8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d740, 0xc00108b320, 0xc00038d730) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1628 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x400f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78b70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78b70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d760, 0xc00108b320, 0xc00038d750) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1629 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ffe) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78ba0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78ba0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d780, 0xc00108b320, 0xc00038d770) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1630 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3fea) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78bd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78bd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d7a0, 0xc00108b320, 0xc00038d790) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1631 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d7c0, 0xc00108b320, 0xc00038d7b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1632 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x4002) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d7e0, 0xc00108b320, 0xc00038d7d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1633 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001369050, 0x3ff4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001369040) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001ae2c80, 0x489e1a0, 0xc000c78c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000c78c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d800, 0xc00108b320, 0xc00038d7f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1634 [semacquire]: sync.runtime_Semacquire(0xc001ae2cf8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc001ae2cf0) /usr/local/go/src/sync/waitgroup.go:130 +0x65 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d820, 0xc00108b320, 0xc00038d810) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1635 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc000c91500, 0x489e1a0, 0xc000c78cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3769 +0x264 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d840, 0xc00108b320, 0xc00038d830) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1636 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc000c91500, 0x489e1a0, 0xc000c78cf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3825 +0x129 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d860, 0xc00108b320, 0xc00038d850) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1637 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x489e1a0, 0xc000c78d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1417 +0xe6 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d8b0, 0xc00108b320, 0xc0002f1f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1638 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000c78f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d8c0, 0xc00108b320, 0xc00156f8b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1639 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000c78f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d8d0, 0xc00108b320, 0xc00156f900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1640 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000c78f60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d8f0, 0xc00108b320, 0xc00156f950) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2373 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c900, 0xc0011a2480, 0xc0016be0c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1642 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x489e1a0, 0xc000c78c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x2ea github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d940, 0xc00108b320, 0xc00038d930) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1643 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000c9a090, 0x9f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000c9a080) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc000839260, 0x489e1a0, 0xc000d01a40, 0xc001095560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x489e1a0, 0xc000d01a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1628 +0x57 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e1a0, 0xc000d01a40, 0xc000321770, 0x25, 0x0, 0x0, 0xc0002f1f40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1644 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x489e1a0, 0xc000c79200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1636 +0x2db github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d960, 0xc00108b320, 0xc0002f1f60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1645 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x489e1a0, 0xc000c79260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:182 +0x181 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d970, 0xc00108b320, 0xc0002f1f80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1646 [select]: github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x489e1a0, 0xc000c79290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:123 +0x148 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d990, 0xc00108b320, 0xc0002f1fa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1647 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x489e1a0, 0xc000c793e0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dab0, 0xc00108b320, 0xc00037c1a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1648 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x489e1a0, 0xc000c79410) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038db10, 0xc00108b320, 0xc00037c1e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1649 [select]: github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc000403360, 0xc00154a3f0, 0xc00038db70, 0x40e5438, 0xc00133c420, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1014 +0x18b github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x489e1a0, 0xc000c791a0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038db80, 0xc00108b320, 0xc000d01ef0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1650 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x489e1a0, 0xc001563a70) /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x107 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038db90, 0xc00108b320, 0xc00037c260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1651 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x489e1a0, 0xc001563b00) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2024 +0x1fc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dc70, 0xc00108b320, 0xc000d01f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1652 [select]: github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x489e1a0, 0xc001563b30) /go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x112 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dc90, 0xc00108b320, 0xc00038dc80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1653 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x489e1a0, 0xc001562480) /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1991 +0x80e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dca0, 0xc00108b320, 0xc00037c320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1654 [select]: github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x489e1a0, 0xc001563bc0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:147 +0x14f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dcf0, 0xc00108b320, 0xc00038dce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1655 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x489e1a0, 0xc001563b60) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:324 +0xfa github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dd00, 0xc00108b320, 0xc00037c380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1656 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x489e1a0, 0xc001563bf0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:682 +0x234 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038dd20, 0xc00108b320, 0xc00037c3a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1657 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x489e1a0, 0xc001563c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:488 +0x479 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038df20, 0xc00108b320, 0xc00156fae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1669 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c790e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da50, 0xc00108b320, 0xc000301ca0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1670 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c79110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da60, 0xc00108b320, 0xc000301cc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1671 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c79140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da70, 0xc00108b320, 0xc000301ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1672 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000c79170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da90, 0xc00108b320, 0xc000301d60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1673 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc000c3cdd0, 0x489e1a0, 0xc000c78fc0, 0xc00108b320, 0x1afed495, 0xed500a8d2, 0x0, 0xc003c61800, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:217 +0x1d1 github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc003c61800, 0xc000d01401) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:279 +0x81 github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc000d01410, 0xc005a4f800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x18d github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x489e1a0, 0xc000c78fc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:277 +0x19b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106daa0, 0xc00108b320, 0xc000301d80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1658 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x489e1a0, 0xc0015c6a80) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000fb8110, 0xc00108b320, 0xc0003ef840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1659 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x489e1a0, 0xc0007350b0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:297 +0x12b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000fb8140, 0xc00108b320, 0xc0015c66f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1660 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x489e1a0, 0xc000cd24e0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:308 +0xda github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000fb8150, 0xc00108b320, 0xc0003ef8a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1661 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x489e1a0, 0xc0015c6a50) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:322 +0xfb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000fb8160, 0xc00108b320, 0xc0015c6720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1662 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x489e1a0, 0xc0015c6a20) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:246 +0x193 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000fb8170, 0xc00108b320, 0xc0003ef8c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1706 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x489e1a0, 0xc0015aae40) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:478 +0x52 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8f70, 0xc0011a2480, 0xc0003ef9e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1968 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0013c9500) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354700, 0xc0011a2480, 0xc0003546f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1707 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x489e1a0, 0xc0015ab080) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f9040, 0xc0011a2480, 0xc0013f9030) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1819 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0012c62c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1964 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0016b7230) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354670, 0xc0011a2480, 0xc000354660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1965 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0013c9710) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354690, 0xc0011a2480, 0xc000354680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1966 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0013c97d0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003546c0, 0xc0011a2480, 0xc0003546a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1967 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0013c9e60) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003546e0, 0xc0011a2480, 0xc0003546d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1963 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc780) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354650, 0xc0011a2480, 0xc000354640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1877 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001481d00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 6783 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003e36cc0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003c290e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0035a01c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2015 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001abfb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2652 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002e58380, 0xc003634000, 0x489e1a0, 0xc00187f6e0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2658 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002a62640, 0xc0003ee850, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002a62640, 0xc0003ee850, 0x5, 0x5, 0xc11b10, 0xc002a4ee00, 0xc0049d9978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000791da0, 0xc0003ee850, 0x5, 0x5, 0x74cff0, 0xc0049d99a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000791da0, 0xc0003ee850, 0x5, 0x5, 0x5, 0xf, 0xc000636900, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001505c00, 0xc0003ee850, 0x5, 0x5, 0x83e500, 0xc0029e69b0, 0xc0049d9a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003ee840, 0x7fffffff, 0x3c06120, 0xc000636900, 0xc0049d9a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003ee840, 0xc001505c00, 0x0, 0x0, 0x7fffffff, 0xc0049d9cb8, 0x4886820, 0x72b56a8, 0x0, 0xc003c85020, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003ee840, 0x7f086d36adf0, 0xc000284fd0, 0xc001505c00, 0x0, 0x0, 0x3de8800, 0xc0017e2780, 0x7fffffff, 0xc0049d9cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc00185df00, 0x3de8800, 0xc0017e2780, 0xc0049d9cb8, 0xc004c9cf50, 0xc0017e2780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00185df00, 0xc00009a070, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0011c1560, 0xc0049d9e00, 0xc0049d9df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0011c1560, 0x3de8800, 0xc0017e2780, 0x2c5e8e0, 0xc000ce8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc00026c060, 0x489e1a0, 0xc001a57320, 0xc00428e580) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc00026c060, 0xc001171a00, 0xc000214870, 0x489e1a0, 0xc001a57320, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc001a57320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c090, 0xc000cfe090, 0xc000791dd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 132407 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x489e1a0, 0xc000ffd1a0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:258 +0x11f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e1a0, 0xc000ffd1a0, 0xc003ca83c0, 0x2a, 0x0, 0x0, 0xc005ef59e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1824 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc002d65ed0, 0x1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc002d65ec0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc001a51820, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:437 +0x89 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc001ab9000, 0x489e0e0, 0xc002d65f00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1191 +0x52 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc001ab9000, 0x489e0e0, 0xc002d65f00, 0xc001636e80, 0x5400, 0x15000, 0xc001636f18, 0xc0018115b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0018fe000, 0x489e0e0, 0xc002d65f00, 0xc001ab9000, 0x5400, 0x15000, 0xc001636f18, 0xc0018115b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444 +0xce github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0016d9877, 0xc0017618a0, 0x489e0e0, 0xc002d65f00, 0xc0018115b0, 0xc0018fe000, 0xc001a51200, 0x48c0ea0, 0xc001761880, 0xc0025e0360, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584 +0x21c created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:513 +0x17b goroutine 2653 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc003632050, 0xc002c68cd0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003632050, 0xc002c68cd0, 0x5, 0x5, 0xc11b10, 0xc0035ec3e0, 0xc004991978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00187f800, 0xc002c68cd0, 0x5, 0x5, 0x74cff0, 0xc0049919a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00187f800, 0xc002c68cd0, 0x5, 0x5, 0x5, 0xf, 0xc000e8fda0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000177800, 0xc002c68cd0, 0x5, 0x5, 0x83e500, 0xc004d39500, 0xc004991a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc002c68cc0, 0x7fffffff, 0x3c06120, 0xc000e8fda0, 0xc004991a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc002c68cc0, 0xc000177800, 0x0, 0x0, 0x7fffffff, 0xc004991cb8, 0x4886820, 0x72b56a8, 0x0, 0xc002d4e2a0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc002c68cc0, 0x7f086d36adf0, 0xc000284fd0, 0xc000177800, 0x0, 0x0, 0x3de8800, 0xc0013d9180, 0x7fffffff, 0xc004991cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0015a7980, 0x3de8800, 0xc0013d9180, 0xc004991cb8, 0xc0039f9cc0, 0xc0013d9180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0015a7980, 0xc000092f70, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc003634000, 0xc004991e00, 0xc004991df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc003634000, 0x3de8800, 0xc0013d9180, 0x2c5e8e0, 0xc001f42700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc001a9d720, 0x489e1a0, 0xc00187f890, 0xc006fbec40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc001a9d720, 0xc001171900, 0xc000214960, 0x489e1a0, 0xc00187f890, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc00187f890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9d740, 0xc0011a2480, 0xc00187f830) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2175 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0032be000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1815 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00173e7c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0025f65a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000358c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1814 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31aab8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0003fbe18, 0x72, 0x4900, 0x49ae, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0003fbe00, 0xc004e6aa80, 0x49ae, 0x49ae, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0003fbe00, 0xc004e6aa80, 0x49ae, 0x49ae, 0x203001, 0x0, 0x49a1) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002270078, 0xc004e6aa80, 0x49ae, 0x49ae, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0069d9720, 0xc004e6aa80, 0x49ae, 0x49ae, 0x76, 0x482c600, 0xc0025f9a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000aad3d8, 0x482c340, 0xc0069d9720, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000aad180, 0x48317e0, 0xc002270078, 0x5, 0xc002270078, 0xc006f30210) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000aad180, 0x40e5100, 0xc000aad2b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000aad180, 0xc00125a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0025f64e0, 0xc0003671b8, 0x9, 0x9, 0xc0050b8358, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0025f64e0, 0xc0003671b8, 0x9, 0x9, 0x9, 0xc0054d13e0, 0x485e320, 0xc006d99340) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003671b8, 0x9, 0x9, 0x482c140, 0xc0025f64e0, 0x0, 0xc000000000, 0xc0069b3300, 0x6f30100) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000367180, 0xc006f301e0, 0xc006f301e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000358c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1972 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc000b3b880, 0xc001123440, 0x489e1a0, 0xc000867860) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1872 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214870, 0x48fa580, 0xc000335050, 0x485c120, 0xc000214870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc000bff140, 0x0, 0xc00133aae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc000bff140, 0xc0003bfa00, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc000bff140, 0xc0003bfa00, 0x40e1ba8, 0x489e1a0, 0xc000bc29f0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc000fae900, 0xc0029ea600, 0xc000aa7500, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc000fae900, 0xc0029ea600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002b2700, 0xc000ddc180, 0x48fef60, 0xc000fae900, 0xc0029ea600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1755 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001772700, 0xc0015d70e0, 0x489e1a0, 0xc000bc2660) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1971 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214870, 0x2, 0xc001752a80, 0xc000207180, 0x48fa520, 0xc0004233e0, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001185f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000423350, 0xc000cfe090, 0xc001185ef0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1756 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000def450, 0xc0003bf890, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000def450, 0xc0003bf890, 0x5, 0x5, 0xc11b10, 0xc000304aa0, 0xc0043d3978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000bc2870, 0xc0003bf890, 0x5, 0x5, 0x74cff0, 0xc0043d39a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000bc2870, 0xc0003bf890, 0x5, 0x5, 0x5, 0xf, 0xc000079c20, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00288b500, 0xc0003bf890, 0x5, 0x5, 0x83e500, 0xc0030977f0, 0xc0043d3a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003bf880, 0x7fffffff, 0x3c06120, 0xc000079c20, 0xc0043d3a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003bf880, 0xc00288b500, 0x0, 0x0, 0x7fffffff, 0xc0043d3cb8, 0x4886820, 0x72b56a8, 0x0, 0xc0002db540, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003bf880, 0x7f086d36adf0, 0xc000284fd0, 0xc00288b500, 0x0, 0x0, 0x3de8800, 0xc003e8ca40, 0x7fffffff, 0xc0043d3cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001ae3a00, 0x3de8800, 0xc003e8ca40, 0xc0043d3cb8, 0xc003d1b8b0, 0xc003e8ca40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001ae3a00, 0xc000084f70, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0015d70e0, 0xc0043d3e00, 0xc0043d3df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0015d70e0, 0x3de8800, 0xc003e8ca40, 0x2c5e8e0, 0xc000c91500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc000334fe0, 0x489e1a0, 0xc000bc2930, 0xc006de5b80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc000334fe0, 0xc0004a4d00, 0xc0010e50e0, 0x489e1a0, 0xc000bc2930, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc000bc2930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335000, 0xc00108b320, 0xc000bc28a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2375 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c930, 0xc0011a2480, 0xc0016be100) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1757 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc002a62550, 0xc0003bf9f0, 0x5, 0x5, 0xc00602b600, 0x1f0, 0xc0000a0f00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002a62550, 0xc0003bf9f0, 0x5, 0x5, 0xc11b10, 0xc0018984a0, 0xc0019e7af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001185d70, 0xc0003bf9f0, 0x5, 0x5, 0x72, 0xc0019e7b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001185d70, 0xc0003bf9f0, 0x5, 0x5, 0x5, 0xc0019e7bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0029ea600, 0xc0003bf9f0, 0x5, 0x5, 0x203001, 0x203001, 0x713a95) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003bf9e0, 0x7fffffff, 0x30, 0xc006d06870, 0xc0019e7c78, 0x2c15de0, 0xc0018d7180, 0x7f086d4aefc8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003bf9e0, 0xc0029ea600, 0x0, 0x0, 0x7fffffff, 0xc0019e7e30, 0x4886820, 0x72b56a8, 0x8, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003bf9e0, 0x7f086d36adf0, 0xc000284fd0, 0xc0029ea600, 0x0, 0x0, 0x3de86e0, 0xc007527f40, 0x7fffffff, 0xc0019e7e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc000bff140, 0x3de86e0, 0xc007527f40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc000335050, 0x489e1a0, 0xc000bc2cf0, 0xc0072bcdf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc000335050, 0xc000214870, 0x489e1a0, 0xc000bc2cf0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc000bc2cf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000335070, 0xc000cfe090, 0xc000bc2a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1758 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0010e50e0, 0x48fa580, 0xc000335220, 0x485c120, 0xc0010e50e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc000bff200, 0x0, 0xc00192d200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc000bff200, 0xc0003bfda0, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc000bff200, 0xc0003bfda0, 0x40e1ba8, 0x489e1a0, 0xc000bc3350) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a74c00, 0x48fef60, 0xc000e4cf00, 0xc00288b600, 0xc0013d4330, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc000e4cf00, 0xc00288b600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000339690, 0xc001a74c00, 0x48fef60, 0xc000e4cf00, 0xc00288b600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1759 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000def810, 0xc0003bfd70, 0x5, 0x5, 0x0, 0x0, 0xc0000a0f00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000def810, 0xc0003bfd70, 0x5, 0x5, 0xc11b10, 0xc000304c20, 0xc004e45af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000bc32c0, 0xc0003bfd70, 0x5, 0x5, 0x3a, 0xc004e45b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000bc32c0, 0xc0003bfd70, 0x5, 0x5, 0x5, 0xc004e45bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00288b600, 0xc0003bfd70, 0x5, 0x5, 0x203001, 0x203001, 0x713a95) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003bfd60, 0x7fffffff, 0x30, 0xc006d06750, 0x71cab6, 0xc004e45c50, 0xc000b00000, 0x74bf20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003bfd60, 0xc00288b600, 0x0, 0x0, 0x7fffffff, 0xc004e45e30, 0x4886820, 0x72b56a8, 0x8, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003bfd60, 0x7f086d36adf0, 0xc000284fd0, 0xc00288b600, 0x0, 0x0, 0x3de86e0, 0xc007527c20, 0x7fffffff, 0xc004e45e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc000bff200, 0x3de86e0, 0xc007527c20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc000335220, 0x489e1a0, 0xc000bc3470, 0xc00602b000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc000335220, 0xc0010e50e0, 0x489e1a0, 0xc000bc3470, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc000bc3470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003353a0, 0xc00108b320, 0xc000bc33e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2376 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c960, 0xc0011a2480, 0xc0016be140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1960 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0013c9650) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354580, 0xc0011a2480, 0xc000354560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2374 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc001969e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00031c910, 0xc0011a2480, 0xc0016be0e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2048 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00142edc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2778 [IO wait]: internal/poll.runtime_pollWait(0x7f086d454010, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00140ff18, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00140ff00, 0xc00168c000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00140ff00, 0xc00168c000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcc5d0, 0xc00168c000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00718a620, 0xc00168c000, 0x791, 0x791, 0x4d, 0x482c600, 0xc003674a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001291e58, 0x482c340, 0xc00718a620, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001291c00, 0x48317e0, 0xc000bcc5d0, 0x5, 0xc000bcc5d0, 0x203001) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001291c00, 0x40e5100, 0xc001291d38, 0xc003674d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001291c00, 0xc003748000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002a6e960, 0xc0013cc1f8, 0x9, 0x9, 0xc1c86c, 0xc001028328, 0xc001441260) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002a6e960, 0xc0013cc1f8, 0x9, 0x9, 0x9, 0x1b, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013cc1f8, 0x9, 0x9, 0x482c140, 0xc002a6e960, 0x0, 0x1b00000000, 0x100000439, 0x4390000001b) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013cc1c0, 0xc0014411d0, 0xc0014411d0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0010281c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1679 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001480100, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00192d560, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000e4cf00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1680 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000e4cf00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1681 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a9e8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00028e118, 0x72, 0x6200, 0x6259, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00028e100, 0xc002074a80, 0x6259, 0x6259, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00028e100, 0xc002074a80, 0x6259, 0x6259, 0xc002a3a938, 0x6201, 0xc000d66180) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0025e4000, 0xc002074a80, 0x6259, 0x6259, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0025f6000, 0xc002074a80, 0x6259, 0x6259, 0xc002074a8d, 0xb4, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc007527b80, 0xc002074a80, 0x6259, 0x6259, 0xd1, 0x482c600, 0xc002a3aa50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000d66258, 0x482c340, 0xc007527b80, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000d66000, 0x7f086d36ae18, 0xc0025f6000, 0x5, 0xc0025f6000, 0xb4) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000d66000, 0x40e5100, 0xc000d66138, 0x10) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000d66000, 0xc001248000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc00192d500, 0xc0003d2e38, 0x9, 0x9, 0xc0017e3988, 0xc004d4e600, 0x51) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc00192d500, 0xc0003d2e38, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003d2e38, 0x9, 0x9, 0x482c140, 0xc00192d500, 0x0, 0x700000000, 0xc00288b600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003d2e00, 0xc001115140, 0xc001115140, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000e4cf00, 0xc001115260, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc001a74c00, 0x48fef60, 0xc000e4cf00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc001a74c00, 0x48fef60, 0xc000e4cf00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2965 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc000d2b500, 0xc0016f4b40, 0x489e1a0, 0xc00190c450) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2784 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001683e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2783 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002ed9100, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002a6fd40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001683e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1691 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc0017d3ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000423780, 0xc000cfe090, 0xc0018a4300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1878 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000cf1040, 0xc00189b500, 0xc000243dc0, 0xf, 0xc000b12c60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0017ac0f0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9ca00, 0xc000cfe090, 0xc002941900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2588 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0034ee440, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0027a1b60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0035a0000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1889 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214870, 0x2, 0xc00189b4a0, 0xc001170640, 0x48fa520, 0xc00172f1a0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc0011ee480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00172f140, 0xc000cfe090, 0xc0011ee450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2104 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001166a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2041 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc0010cc840) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d380, 0xc0011a2480, 0xc00037dd40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2042 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0018307d0, 0xc00037df10, 0x5, 0x5, 0xba, 0xc0026d8180, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0018307d0, 0xc00037df10, 0x5, 0x5, 0xc11b10, 0xc0002afe20, 0xc0023f7858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0011085a0, 0xc00037df10, 0x5, 0x5, 0xc007afc080, 0xc0023f7880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0011085a0, 0xc00037df10, 0x5, 0x5, 0x5, 0xc0026a8d80, 0xc0023f7948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0018d5700, 0xc00037df10, 0x5, 0x5, 0x40, 0x40, 0xc0018d7180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00037df00, 0x7fffffff, 0xc0e5c1, 0xc00133c300, 0x58, 0x58, 0xc0021c6060, 0xc001158060) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00037df00, 0xc0018d5700, 0x0, 0x0, 0x7fffffff, 0xc0023f7b98, 0x4886820, 0x72b56a8, 0x0, 0xc007bfe005, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00037df00, 0x7f086d36adf0, 0xc000284fd0, 0xc0018d5700, 0x0, 0x0, 0x3de9580, 0xc007ba60c0, 0x7fffffff, 0xc0023f7b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0018f6c00, 0x3de9580, 0xc007ba60c0, 0xc0023f7b98, 0x0, 0xc007ba60c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0018f6c00, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00196c480, 0xc0023f7ce0, 0xc0023f7cd0, 0x72b56a8, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00196c480, 0x3de9580, 0xc007ba60c0, 0xc007afc040, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc00038d430, 0xc007afc040, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc0011084e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d3c0, 0xc000cfe090, 0xc00037dda0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2178 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001a84a80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002dc0720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001abfc80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 7201 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001697200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2044 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc001108630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d440, 0xc000cfe090, 0xc00037df20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2045 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc000ddfc70, 0x489e1a0, 0xc001108750, 0xc0025edb00, 0x48f9ec0, 0xc0003031c0, 0xc0011a2480, 0xc0018963a4, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:366 +0x344 github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x489e1a0, 0xc001108750) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:125 +0x3c2 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d590, 0xc0011a2480, 0xc0014442c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2043 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc000b3b880, 0xc00196c480, 0x489e1a0, 0xc001108480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2047 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00071e340, 0xc00294fe00, 0xc0002bea40, 0xf, 0xc00245df20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0011088d0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d760, 0xc0011a2480, 0xc001830be0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1776 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002c20000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2046 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0014446c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1959 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0025edda0, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0025edd90) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc0025edc80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 2014 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001a84740, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002805ec0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001abfb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1962 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc690) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354630, 0xc0011a2480, 0xc0003545c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1958 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0010e50e0, 0x1, 0xc0012bd560, 0xc0004a4d00, 0x48fa520, 0xc000334fe0, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc000bc2570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f9870, 0xc00108b320, 0xc0005c2240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1969 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc6f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354720, 0xc0011a2480, 0xc000354710) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1973 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002a62690, 0xc0003cd0b0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002a62690, 0xc0003cd0b0, 0x5, 0x5, 0xc11b10, 0xc0018984c0, 0xc001387978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000682480, 0xc0003cd0b0, 0x5, 0x5, 0x74cff0, 0xc0013879a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000682480, 0xc0003cd0b0, 0x5, 0x5, 0x5, 0xf, 0xc000636900, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0029ea700, 0xc0003cd0b0, 0x5, 0x5, 0x83e500, 0xc006250160, 0xc001387a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003cd0a0, 0x7fffffff, 0x3c06120, 0xc000636900, 0xc001387a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003cd0a0, 0xc0029ea700, 0x0, 0x0, 0x7fffffff, 0xc001387cb8, 0x4886820, 0x72b56a8, 0x0, 0xc006bb2160, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003cd0a0, 0x7f086d36adf0, 0xc000284fd0, 0xc0029ea700, 0x0, 0x0, 0x3de8800, 0xc0054630c0, 0x7fffffff, 0xc001387cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001585800, 0x3de8800, 0xc0054630c0, 0xc001387cb8, 0xc006d58320, 0xc0054630c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001585800, 0xc00008e570, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001123440, 0xc001387e00, 0xc001387df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001123440, 0x3de8800, 0xc0054630c0, 0x2c5e8e0, 0xc000ce8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0004233e0, 0x489e1a0, 0xc000682540, 0xc0024b9f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0004233e0, 0xc000207180, 0xc000214870, 0x489e1a0, 0xc000682540, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc000682540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000423400, 0xc000cfe090, 0xc0006824e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2655 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc003632230, 0xc002c68eb0, 0x5, 0x5, 0x0, 0x0, 0xc000095400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003632230, 0xc002c68eb0, 0x5, 0x5, 0xc11b10, 0xc0035ec400, 0xc001a17af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00187fa10, 0xc002c68eb0, 0x5, 0x5, 0x39, 0xc001a17b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00187fa10, 0xc002c68eb0, 0x5, 0x5, 0x5, 0xc001a17bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000177900, 0xc002c68eb0, 0x5, 0x5, 0x203001, 0x203001, 0x713a95) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc002c68ea0, 0x7fffffff, 0x732a2f, 0xc000e88480, 0x4, 0xc001a17c70, 0x733ecb, 0xc000ddade0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc002c68ea0, 0xc000177900, 0x0, 0x0, 0x7fffffff, 0xc001a17e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc002c68ea0, 0x7f086d36adf0, 0xc000284fd0, 0xc000177900, 0x0, 0x0, 0x3de86e0, 0xc006de2c80, 0x7fffffff, 0xc001a17e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00190a9c0, 0x3de86e0, 0xc006de2c80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc001a9d7f0, 0x489e1a0, 0xc00187fad0, 0xc00708b800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc001a9d7f0, 0xc000214870, 0x489e1a0, 0xc00187fad0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc00187fad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9d810, 0xc000cfe090, 0xc00187faa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2173 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a368, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc002dc4618, 0x72, 0x4300, 0x4320, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc002dc4600, 0xc00317a000, 0x4320, 0x4320, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc002dc4600, 0xc00317a000, 0x4320, 0x4320, 0x203001, 0x0, 0x4313) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00043c2d8, 0xc00317a000, 0x4320, 0x4320, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0077c4700, 0xc00317a000, 0x4320, 0x4320, 0x59, 0x482c600, 0xc0025f8a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002e58958, 0x482c340, 0xc0077c4700, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002e58700, 0x48317e0, 0xc00043c2d8, 0x5, 0xc00043c2d8, 0x3c) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002e58700, 0x40e5100, 0xc002e58838, 0xc0025f8d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002e58700, 0xc002ec0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0017537a0, 0xc0013e2738, 0x9, 0x9, 0xc1c86c, 0xc001029668, 0xc0011c2ff0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0017537a0, 0xc0013e2738, 0x9, 0x9, 0x9, 0x33, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013e2738, 0x9, 0x9, 0x482c140, 0xc0017537a0, 0x0, 0x3300000000, 0x10000004b, 0x4b00000033) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013e2700, 0xc0011c2f60, 0xc0011c2f60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001029500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2174 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002ed8000, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002dc0d80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001029500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2176 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a028, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00140e498, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00140e480, 0xc0016ea800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00140e480, 0xc0016ea800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00109c038, 0xc0016ea800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc006eb97a0, 0xc0016ea800, 0x791, 0x791, 0x76, 0x482c600, 0xc002f3da78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001118258, 0x482c340, 0xc006eb97a0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001118000, 0x48317e0, 0xc00109c038, 0x5, 0xc00109c038, 0xc006842900) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001118000, 0x40e5100, 0xc001118138, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001118000, 0xc0032dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0028e0540, 0xc0013e2498, 0x9, 0x9, 0xc006500298, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0028e0540, 0xc0013e2498, 0x9, 0x9, 0x9, 0xc0048deae0, 0x485e320, 0xc0050e5100) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013e2498, 0x9, 0x9, 0x482c140, 0xc0028e0540, 0x0, 0xc000000000, 0xc0064de100, 0x6842800) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013e2460, 0xc0068428d0, 0xc0068428d0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0032be000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2177 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001684200, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0028e0600, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0032be000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 10184 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003c2a000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1989 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc0002f0f20, 0x7f086677d630, 0xc00031dee0, 0x7f086677d630, 0xc00031dee0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc0002f0f20, 0x48fa760, 0xc00031dee0, 0xc02f00010301, 0xc00031dee0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc00037da20, 0x48eb940, 0xc001269200, 0x0, 0xc00192d200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc00037da20, 0x48eb940, 0xc001269200, 0xc0002b0620, 0x40e1e60, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc00037da20, 0x48eb940, 0xc001269200, 0xc0002b0620, 0x40e1e60, 0x489e1a0, 0xc0010cca50) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a74c00, 0x48fef60, 0xc000e4cf00, 0xc001934100, 0xc0013d4b40, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc000e4cf00, 0xc001934100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000339690, 0xc001a74c00, 0x48fef60, 0xc000e4cf00, 0xc001934100) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2012 [select]: net.(*pipe).read(0xc001fc6400, 0xc00188d800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc001fc6400, 0xc00188d800, 0x791, 0x791, 0xc00188d80d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc006bebec0, 0xc00188d800, 0x791, 0x791, 0x2e, 0x482c600, 0xc001826ac0) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0013ef058, 0x482c340, 0xc006bebec0, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0013eee00, 0x7f086d451a50, 0xc001fc6400, 0x5, 0xc001fc6400, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0013eee00, 0x40e5100, 0xc0013eef38, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0013eee00, 0xc002da8000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002805c20, 0xc0010fa118, 0x9, 0x9, 0x6fe9a9, 0xc002c04180, 0xc001826e40) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002805c20, 0xc0010fa118, 0x9, 0x9, 0x9, 0x0, 0xc00170f978, 0x74a6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0010fa118, 0x9, 0x9, 0x482c140, 0xc002805c20, 0x0, 0xc000000000, 0xc001826ea7, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0010fa0e0, 0xc002cef040, 0xc002cef040, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0016f6380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2016 [select]: net.(*pipe).read(0xc001fc6380, 0xc0016ae800, 0x75b, 0x75b, 0x203001, 0x0, 0x74e) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc001fc6380, 0xc0016ae800, 0x75b, 0x75b, 0xc0016ae80d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc006bebee0, 0xc0016ae800, 0x75b, 0x75b, 0x2e, 0x482c600, 0xc001606a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00176a958, 0x482c340, 0xc006bebee0, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00176a700, 0x7f086d451a50, 0xc001fc6380, 0x5, 0xc001fc6380, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00176a700, 0x40e5100, 0xc00176a838, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00176a700, 0xc002d90000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002805b00, 0xc0010fa038, 0x9, 0x9, 0xc00439e000, 0x7f086d499b28, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002805b00, 0xc0010fa038, 0x9, 0x9, 0x9, 0xbdd055, 0xc002cef06c, 0xc003b1f920) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0010fa038, 0x9, 0x9, 0x482c140, 0xc002805b00, 0x0, 0x0, 0xc002cef060, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0010fa000, 0xc002cef060, 0xc002cef060, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001abfb00, 0xc001031080, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0025ed980, 0x48fef60, 0xc001abfb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0025ed980, 0x48fef60, 0xc001abfb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2017 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00188e3c0, 0xc002d62330, 0x5, 0x5, 0xbd, 0xc005ec8290, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00188e3c0, 0xc002d62330, 0x5, 0x5, 0xc11b10, 0xc0015d8080, 0xc002dcb858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0011bc300, 0xc002d62330, 0x5, 0x5, 0xc006de30c0, 0xc002dcb880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0011bc300, 0xc002d62330, 0x5, 0x5, 0x5, 0xc0010f1c80, 0xc002dcb948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002dce000, 0xc002d62330, 0x5, 0x5, 0x40, 0x40, 0xc003813c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc002d62320, 0x7fffffff, 0xc0e5c1, 0xc0011657a0, 0x58, 0x58, 0xc006f4c540, 0xc00106d910) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc002d62320, 0xc002dce000, 0x0, 0x0, 0x7fffffff, 0xc002dcbb98, 0x4886820, 0x72b56a8, 0x0, 0xc0029cce75, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc002d62320, 0x7f086d36adf0, 0xc000284fd0, 0xc002dce000, 0x0, 0x0, 0x3de9580, 0xc0076070b0, 0x7fffffff, 0xc002dcbb98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc002dc4000, 0x3de9580, 0xc0076070b0, 0xc002dcbb98, 0x0, 0xc0076070b0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc002dc4000, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001aee000, 0xc002dcbce0, 0xc002dcbcd0, 0x72b56a8, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001aee000, 0x3de9580, 0xc0076070b0, 0xc006de3080, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0018100e0, 0xc006de3080, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc0011bc240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b4eb0, 0xc00108b320, 0xc0003b5560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2179 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001abfc80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2018 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc720) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354740, 0xc0011a2480, 0xc000354730) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2019 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc750) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354760, 0xc0011a2480, 0xc000354750) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2020 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc6c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354780, 0xc0011a2480, 0xc000354770) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2021 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc7e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003547a0, 0xc0011a2480, 0xc000354790) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2022 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001a56db0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003547c0, 0xc0011a2480, 0xc0003547b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2023 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0010cc7b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003547e0, 0xc0011a2480, 0xc0003547d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2024 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0016f50e0, 0x489e0e0, 0xc001685bc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 2025 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0016f5320, 0x489e0e0, 0xc001685bc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 2026 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x489e1a0, 0xc001a575c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1716 +0x370 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355050, 0xc0011a2480, 0xc0010ac810) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2027 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x489e1a0, 0xc001a57650) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1797 +0x257 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003550c0, 0xc0011a2480, 0xc0003550b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2028 [select]: github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x40e5438, 0xc00114e780, 0xc002288c40, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11b created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x105 goroutine 2029 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc001a57680) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003551f0, 0xc0011a2480, 0xc0016be600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2030 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x489e1a0, 0xc001a576b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1751 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355250, 0xc0011a2480, 0xc0016be640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2031 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a918, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001fc6198, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001fc6180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001fc6180, 0xc000466380, 0x7f086d4ac718, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000bcc748, 0x71f47f, 0xc000ffe170, 0xc000d22ef0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000bcc748, 0x40e2688, 0xc001685d40, 0xc001896250, 0x4906de0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc001685d40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x489e1a0, 0xc0016b77d0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1764 +0x2f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355270, 0xc0011a2480, 0xc0016be980) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2032 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 net/http.(*Server).Serve(0xc000ddf930, 0x4886ea0, 0xc0016be940, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x489e1a0, 0xc0016b7800) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1780 +0x127 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355280, 0xc0011a2480, 0xc0010ac990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2033 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 crypto/tls.(*listener).Accept(0xc0016be9a0, 0x3d2f600, 0xc001a578f0, 0x3a88520, 0x6f95560) /usr/local/go/src/crypto/tls/tls.go:62 +0x37 net/http.(*Server).Serve(0xc001fc16c0, 0x486e6e0, 0xc0016be9a0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x489e1a0, 0xc001a57860) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1791 +0x3f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355310, 0xc0011a2480, 0xc0016be9c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2050 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x489e1a0, 0xc000c02c90) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1654 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355370, 0xc0011a2480, 0xc0016bea40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2051 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x489e1a0, 0xc0016b77a0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1706 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355390, 0xc0011a2480, 0xc001685dc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2052 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc0025ed980, 0x4886ea0, 0xc0016bea60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x1e9 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x489e1a0, 0xc001a57710) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x46 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003553b0, 0xc0011a2480, 0xc0016bea80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2053 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x489e1a0, 0xc001a576e0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1198 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003554d0, 0xc0011a2480, 0xc0010acde0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2013 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001a84780, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002805da0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0016f6380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2055 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc002c06080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2056 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x489e1a0, 0xc0010cc810) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1228 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355600, 0xc0011a2480, 0xc0016bee40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1990 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000c9a090, 0xa0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000c9a080) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc000839260, 0x489e1a0, 0xc0010cca50, 0xc00295efc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc0010cca50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e1a0, 0xc0010cca50, 0xc000401ca0, 0x1d, 0x0, 0x0, 0xc0002b0680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2057 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc002c04a20, 0xc000001d0d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc002c04a10) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc002c04900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 2058 [select]: github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x489e1a0, 0xc000dec2d0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:44 +0x16b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355890, 0xc0011a2480, 0xc0010acf60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2059 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x489e1a0, 0xc000dec120) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355aa0, 0xc0011a2480, 0xc0016bfa00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2060 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x489e1a0, 0xc001108660) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1365 +0x3b4 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355ad0, 0xc0011a2480, 0xc000355ab0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2061 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x489e1a0, 0xc000dec300) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1400 +0x33c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355af0, 0xc0011a2480, 0xc000355ae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2062 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a848, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001fc6218, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001fc6200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001fc6200, 0xc000c74101, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000bcc760, 0xc001829e80, 0xc001829e88, 0x28) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000bcc760, 0x40e2690, 0xc001685d80, 0x48faa00, 0xc000c74150) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc001685d80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1() /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1731 +0x2f sync.(*Once).Do(0xc0003f6520, 0xc002c2df50) /usr/local/go/src/sync/once.go:44 +0xb3 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc0010ccab0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355b10, 0xc0011a2480, 0xc0016bfa20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1981 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002d5c000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2121 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001166c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2088 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00071e340, 0xc002c36ea0, 0xc002a43750, 0xf, 0xc0024813e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001109c80) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038de40, 0xc0011a2480, 0xc001831040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2105 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a6a8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0018f6e18, 0x72, 0x3c00, 0x3cd4, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0018f6e00, 0xc002f6e000, 0x3cd4, 0x3cd4, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0018f6e00, 0xc002f6e000, 0x3cd4, 0x3cd4, 0x203002, 0x0, 0x3cc7) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110438, 0xc002f6e000, 0x3cd4, 0x3cd4, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc008170800, 0xc002f6e000, 0x3cd4, 0x3cd4, 0x6f, 0x482c600, 0xc00182aa78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002c325d8, 0x482c340, 0xc008170800, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002c32380, 0x48317e0, 0xc000110438, 0x5, 0xc000110438, 0x52) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002c32380, 0x40e5100, 0xc002c324b8, 0xc00182ad40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002c32380, 0xc002c88000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001303ce0, 0xc00295bb58, 0x9, 0x9, 0xc1c86c, 0xc001166be8, 0xc0010adb60) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001303ce0, 0xc00295bb58, 0x9, 0x9, 0x9, 0x49, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00295bb58, 0x9, 0x9, 0x482c140, 0xc001303ce0, 0x0, 0x4900000000, 0x100000003, 0x300000049) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00295bb20, 0xc0010ada40, 0xc0010ada40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001166a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2124 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002c06f00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0025f71a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc002c43200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2087 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001445240) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2151 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc002d65180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2106 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002c06240, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002c36d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001166a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2107 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0011e5ec0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002c30900, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0026a9500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2108 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0026a9500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2109 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a5d8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000eaf598, 0x72, 0x2500, 0x25cf, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000eaf580, 0xc00272e600, 0x25cf, 0x25cf, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000eaf580, 0xc00272e600, 0x25cf, 0x25cf, 0xc001602938, 0x2501, 0xc0013ef300) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002a51718, 0xc00272e600, 0x25cf, 0x25cf, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002c30360, 0xc00272e600, 0x25cf, 0x25cf, 0xc00272e60d, 0xc3, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc006bcc5e0, 0xc00272e600, 0x25cf, 0x25cf, 0xe0, 0x482c600, 0xc001602a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0013ef3d8, 0x482c340, 0xc006bcc5e0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0013ef180, 0x7f086d36ae18, 0xc002c30360, 0x5, 0xc002c30360, 0xc3) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0013ef180, 0x40e5100, 0xc0013ef2b8, 0x7f08667a08e0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0013ef180, 0xc002c70000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002c307e0, 0xc001404ff8, 0x9, 0x9, 0xc001a83b88, 0xc006360000, 0xba) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002c307e0, 0xc001404ff8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc001404ff8, 0x9, 0x9, 0x482c140, 0xc002c307e0, 0x0, 0x300000000, 0xc001934200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc001404fc0, 0xc0015c6120, 0xc0015c6120, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0026a9500, 0xc0010adce0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc0026a9500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc0026a9500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1991 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc000c9d000, 0x48f9f20, 0xc00111e040, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:193 +0x625 github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x3e992e0, 0xc000c9d000, 0x48eb940, 0xc0012692c0, 0x0, 0xc002c30780) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:569 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3e992e0, 0xc000c9d000, 0x48eb940, 0xc0012692c0, 0xc0002b09c0, 0x40df160, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3e992e0, 0xc000c9d000, 0x48eb940, 0xc0012692c0, 0xc0002b09c0, 0x40df160, 0x489e1a0, 0xc0010cccc0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc0026a9500, 0xc001934200, 0xc000aa6ab0, 0x6f9bba0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc0026a9500, 0xc001934200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001992830, 0xc000ddc180, 0x48fef60, 0xc0026a9500, 0xc001934200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1992 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc00273ae60, 0xc0002b09b0, 0x5, 0x5, 0x70a17f, 0x7f0865bc16b0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00273ae60, 0xc0002b09b0, 0x5, 0x5, 0xc11b10, 0xc000401d20, 0xc005d27888) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0010ccc60, 0xc0002b09b0, 0x5, 0x5, 0x6f533a, 0xc005d278b0, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0010ccc60, 0xc0002b09b0, 0x5, 0x5, 0x5, 0xc005d27968, 0xf02137, 0x3a62440) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001934200, 0xc0002b09b0, 0x5, 0x5, 0x7ee108, 0xc0010ccc00, 0x3bbf7c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002b09a0, 0x7fffffff, 0x71fd21, 0x40e43b8, 0xc005d27a08, 0x20, 0xc002558f00, 0x40e43b8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0002b09a0, 0xc001934200, 0x0, 0x0, 0x7fffffff, 0xc005d27bc0, 0x4886820, 0x72b56a8, 0x8, 0x6f8939, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0002b09a0, 0x7f086d36adf0, 0xc000284fd0, 0xc001934200, 0x0, 0x0, 0x3dd1dc0, 0xc005b940a0, 0x7fffffff, 0xc005d27bc0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0012692c0, 0x3dd1dc0, 0xc005b940a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc00111e040, 0x768c8d, 0xc9d048, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:588 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc000c9d000, 0x489e1a0, 0xc0010ccea0, 0xc000011bf8, 0xc001a83cc0, 0xc005d27f50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:334 +0x341 github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1(0x489e1a0, 0xc0010ccea0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:141 +0x9e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00111e070, 0xc000cfe090, 0xc001a83d40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2741 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0014785a0, 0xc000356450, 0x5, 0x5, 0xbe, 0xc001671c88, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0014785a0, 0xc000356450, 0x5, 0x5, 0xc11b10, 0xc00121c1e0, 0xc002e85858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000b58810, 0xc000356450, 0x5, 0x5, 0xc0040374e0, 0xc002e85880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000b58810, 0xc000356450, 0x5, 0x5, 0x5, 0xc000fc7680, 0xc002e85948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000485b00, 0xc000356450, 0x5, 0x5, 0xc004cc0160, 0x40, 0xc00018dc00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000356440, 0x7fffffff, 0xc0e5c1, 0xc002c60b40, 0x58, 0x58, 0xc002316d80, 0xc00198c6c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000356440, 0xc000485b00, 0x0, 0x0, 0x7fffffff, 0xc002e85b98, 0x4886820, 0x72b56a8, 0x40, 0xc006c93435, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000356440, 0x7f086d36adf0, 0xc000284fd0, 0xc000485b00, 0x0, 0x0, 0x3de9580, 0xc00344f3e0, 0x7fffffff, 0xc002e85b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000eae000, 0x3de9580, 0xc00344f3e0, 0xc002e85b98, 0x0, 0xc00344f3e0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000eae000, 0x5, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0017ae000, 0xc002e85ce0, 0xc002e85cd0, 0xc002a0cb80, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0017ae000, 0x3de9580, 0xc00344f3e0, 0xc0040374a0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0013f8110, 0xc0040374a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc000b58750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00106da00, 0xc00108b320, 0xc00030cd00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2395 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00071e340, 0xc002a6f980, 0xc002a437a0, 0xf, 0xc0012cb500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0010cd440) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9cd30, 0xc0011a2480, 0xc002728050) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2126 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a298, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001585c18, 0x72, 0x2700, 0x2738, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001585c00, 0xc001270800, 0x2738, 0x2738, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001585c00, 0xc001270800, 0x2738, 0x2738, 0xc001601938, 0x2701, 0xc0018d6f80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0029ba0d0, 0xc001270800, 0x2738, 0x2738, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc001753080, 0xc001270800, 0x2738, 0x2738, 0xc00127080d, 0x24, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc001f64f80, 0xc001270800, 0x2738, 0x2738, 0x41, 0x482c600, 0xc001601a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0018d7058, 0x482c340, 0xc001f64f80, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0018d6e00, 0x7f086d36ae18, 0xc001753080, 0x5, 0xc001753080, 0x24) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0018d6e00, 0x40e5100, 0xc0018d6f38, 0x4b) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0018d6e00, 0xc002ea8000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002ca4c60, 0xc000219298, 0x9, 0x9, 0xc001252c08, 0xc004cc0020, 0x1b) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002ca4c60, 0xc000219298, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000219298, 0x9, 0x9, 0x482c140, 0xc002ca4c60, 0x0, 0x4b00000000, 0xc003786600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000219260, 0xc001969e60, 0xc001969e60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002c43200, 0xc001969f80, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc002c43200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc002c43200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2125 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002c43200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2123 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002c06780, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002ca4720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001166c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2083 [semacquire]: sync.runtime_SemacquireMutex(0xc0003f6524, 0x7f086d498d00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc0003f6520) /usr/local/go/src/sync/mutex.go:134 +0x109 sync.(*Once).Do(0xc0003f6520, 0xc00179d750) /usr/local/go/src/sync/once.go:40 +0x3b github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc001968600) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038d8a0, 0xc0011a2480, 0xc0017cc700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2122 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a508, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001585918, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001585900, 0xc0010f2000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001585900, 0xc0010f2000, 0x791, 0x791, 0x203000, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110588, 0xc0010f2000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc003b13500, 0xc0010f2000, 0x791, 0x791, 0x8d, 0x482c600, 0xc00160da78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002c32cd8, 0x482c340, 0xc003b13500, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002c32a80, 0x48317e0, 0xc000110588, 0x5, 0xc000110588, 0x70) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002c32a80, 0x40e5100, 0xc002c32bb8, 0xc00160dd40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002c32a80, 0xc002e1e000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002ca4660, 0xc00295bed8, 0x9, 0x9, 0xc1c86c, 0xc001166da8, 0xc001968f90) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002ca4660, 0xc00295bed8, 0x9, 0x9, 0x9, 0x67, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00295bed8, 0x9, 0x9, 0x482c140, 0xc002ca4660, 0x0, 0x6700000000, 0x10000000b, 0xb00000067) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00295bea0, 0xc001968f00, 0xc001968f00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001166c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2180 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a438, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000eaf918, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000eaf900, 0xc0016af000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000eaf900, 0xc0016af000, 0x75b, 0x75b, 0xc00258b938, 0x701, 0xc001fc9680) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002a51898, 0xc0016af000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002ca40c0, 0xc0016af000, 0x75b, 0x75b, 0xc0016af00d, 0x58, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc008170680, 0xc0016af000, 0x75b, 0x75b, 0x75, 0x482c600, 0xc00258ba50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001fc9758, 0x482c340, 0xc008170680, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001fc9500, 0x7f086d36ae18, 0xc002ca40c0, 0x5, 0xc002ca40c0, 0x58) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001fc9500, 0x40e5100, 0xc001fc9638, 0xb) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001fc9500, 0xc002e06000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002dc05a0, 0xc0010fa2d8, 0x9, 0x9, 0xc003bd7488, 0xc0028ba230, 0x4f) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002dc05a0, 0xc0010fa2d8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0010fa2d8, 0x9, 0x9, 0x482c140, 0xc002dc05a0, 0x0, 0xb00000000, 0xc003bbae00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0010fa2a0, 0xc0010318f0, 0xc0010318f0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001abfc80, 0xc001031a10, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000ddc180, 0x48fef60, 0xc001abfc80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000ddc180, 0x48fef60, 0xc001abfc80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2453 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000b87400, 0xc0002c5850, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000b87400, 0xc0002c5850, 0x5, 0x5, 0xc11b10, 0xc00133ee20, 0xc0048b9af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0011ee900, 0xc0002c5850, 0x5, 0x5, 0x0, 0xc0048b9b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0011ee900, 0xc0002c5850, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0015c4600, 0xc0002c5850, 0x5, 0x5, 0x0, 0x200000002, 0x2) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002c5840, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0002c5840, 0xc0015c4600, 0x0, 0x0, 0x7fffffff, 0xc0048b9e30, 0x4886820, 0x72b56a8, 0x8, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0002c5840, 0x7f086d36adf0, 0xc000284fd0, 0xc0015c4600, 0x0, 0x0, 0x3de86e0, 0xc007d745e0, 0x7fffffff, 0xc0048b9e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001550cc0, 0x3de86e0, 0xc007d745e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc00172f270, 0x489e1a0, 0xc0011ee9c0, 0xc0077b9000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc00172f270, 0xc0010e50e0, 0x489e1a0, 0xc0011ee9c0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0011ee9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00172f290, 0xc00108b320, 0xc0011ee990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2452 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0010e50e0, 0x48fa580, 0xc00172f270, 0x485c120, 0xc0010e50e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc001550cc0, 0x0, 0xc00295fbc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc001550cc0, 0xc0002c5860, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc001550cc0, 0xc0002c5860, 0x40e1ba8, 0x489e1a0, 0xc0011ee960) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a74c00, 0x48fef60, 0xc0018d2c00, 0xc0015c4600, 0xc0013d4330, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc0018d2c00, 0xc0015c4600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002b2f10, 0xc001a74c00, 0x48fef60, 0xc0018d2c00, 0xc0015c4600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2450 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc000d2bc00, 0xc001435680, 0x489e1a0, 0xc0011ee540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2420 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00148a080, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002cf61e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001028000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2419 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a1c8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001821e98, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001821e80, 0xc000b0e800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001821e80, 0xc000b0e800, 0x791, 0x791, 0x203001, 0x20300100000000, 0x7f084e7e3fff) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0025e4188, 0xc000b0e800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc007233ea0, 0xc000b0e800, 0x791, 0x791, 0x70a86e, 0x7f084e745f00, 0xc002783a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003004258, 0x482c340, 0xc007233ea0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003004000, 0x48317e0, 0xc0025e4188, 0x5, 0xc0025e4188, 0xc004b50510) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003004000, 0x40e5100, 0xc003004138, 0xc000b00000) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003004000, 0xc001f04000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002cf60c0, 0xc00028c498, 0x9, 0x9, 0xc00777d318, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002cf60c0, 0xc00028c498, 0x9, 0x9, 0x9, 0xc002ca7e90, 0x485e320, 0xc0053a4180) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00028c498, 0x9, 0x9, 0x482c140, 0xc002cf60c0, 0x0, 0xc000000000, 0xc00725fc00, 0x4b50400) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00028c460, 0xc004b504e0, 0xc004b504e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001028000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2451 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000b87130, 0xc0002c55f0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000b87130, 0xc0002c55f0, 0x5, 0x5, 0xc11b10, 0xc00133ede0, 0xc005395978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0011ee6f0, 0xc0002c55f0, 0x5, 0x5, 0x74cff0, 0xc0053959a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0011ee6f0, 0xc0002c55f0, 0x5, 0x5, 0x5, 0xf, 0xc000636900, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0015c4500, 0xc0002c55f0, 0x5, 0x5, 0x83e500, 0xc00666a8b0, 0xc005395a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002c55e0, 0x7fffffff, 0x3c06120, 0xc000636900, 0xc005395a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0002c55e0, 0xc0015c4500, 0x0, 0x0, 0x7fffffff, 0xc005395cb8, 0x4886820, 0x72b56a8, 0x0, 0xc0072a25c0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0002c55e0, 0x7f086d36adf0, 0xc000284fd0, 0xc0015c4500, 0x0, 0x0, 0x3de8800, 0xc003444900, 0x7fffffff, 0xc005395cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001ae2b80, 0x3de8800, 0xc003444900, 0xc005395cb8, 0xc007ad43c0, 0xc003444900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001ae2b80, 0xc00008c070, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001435680, 0xc005395e00, 0xc005395df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001435680, 0x3de8800, 0xc003444900, 0x2c5e8e0, 0xc000ce8000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc00172f1a0, 0x489e1a0, 0xc0011ee780, 0xc002c06a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc00172f1a0, 0xc001170640, 0xc000214870, 0x489e1a0, 0xc0011ee780, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0011ee780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00172f1c0, 0xc000cfe090, 0xc0011ee720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2785 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453da0, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0010a0198, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0010a0180, 0xc001a80000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0010a0180, 0xc001a80000, 0x75b, 0x75b, 0xc00160f938, 0x701, 0xc000af5680) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000c74408, 0xc001a80000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00228b080, 0xc001a80000, 0x75b, 0x75b, 0xc001a8000d, 0x67, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc001f64fa0, 0xc001a80000, 0x75b, 0x75b, 0x84, 0x482c600, 0xc00160fa50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000af5758, 0x482c340, 0xc001f64fa0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000af5500, 0x7f086d36ae18, 0xc00228b080, 0x5, 0xc00228b080, 0x67) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000af5500, 0x40e5100, 0xc000af5638, 0x7f086d3aeaf8) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000af5500, 0xc003380000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002a6fc80, 0xc0013cc738, 0x9, 0x9, 0xc0018b6ec8, 0xc004bd2000, 0x5e) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002a6fc80, 0xc0013cc738, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013cc738, 0x9, 0x9, 0x482c140, 0xc002a6fc80, 0x0, 0x300000000, 0xc002dce600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013cc700, 0xc0011c3e90, 0xc0011c3e90, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001683e00, 0xc0007901e0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0025ed980, 0x48fef60, 0xc001683e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0025ed980, 0x48fef60, 0xc001683e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1879 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002ffa000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2418 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001028000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2217 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc0007b0310, 0x489e1a0, 0xc0017128d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:134 +0x333 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00111e180, 0xc0011a2480, 0xc00111e170) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2218 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc001712900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00111e380, 0xc0011a2480, 0xc0017ccb20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2183 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x489e1a0, 0xc001969110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:151 +0x4a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e3d0, 0xc0011a2480, 0xc002d62700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2184 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e3f0, 0xc0011a2480, 0xc00102e3e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2185 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e410, 0xc0011a2480, 0xc00102e400) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2186 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eee) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019691a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019691a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e430, 0xc0011a2480, 0xc00102e420) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2187 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019691d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019691d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e450, 0xc0011a2480, 0xc00102e440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2188 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e480, 0xc0011a2480, 0xc00102e460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2189 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ecf) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e500, 0xc0011a2480, 0xc00102e4d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2190 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e520, 0xc0011a2480, 0xc00102e510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2191 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019692c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019692c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e540, 0xc0011a2480, 0xc00102e530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2192 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eca) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019692f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019692f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e560, 0xc0011a2480, 0xc00102e550) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2193 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ece) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e580, 0xc0011a2480, 0xc00102e570) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2226 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3edb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e5a0, 0xc0011a2480, 0xc00102e590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2227 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eea) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e600, 0xc0011a2480, 0xc00102e5d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2228 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3edd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019693b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019693b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e630, 0xc0011a2480, 0xc00102e610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2229 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e690, 0xc0011a2480, 0xc00102e650) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2230 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e940, 0xc0011a2480, 0xc00102e900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2231 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e9a0, 0xc0011a2480, 0xc00102e950) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2232 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019694a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019694a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e9d0, 0xc0011a2480, 0xc00102e9b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2233 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019694d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019694d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102e9f0, 0xc0011a2480, 0xc00102e9e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2234 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ea10, 0xc0011a2480, 0xc00102ea00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2235 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ea80, 0xc0011a2480, 0xc00102ea70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2236 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ecd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019690e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019690e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eb10, 0xc0011a2480, 0xc00102eb00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2237 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019695f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019695f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eb40, 0xc0011a2480, 0xc00102eb30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2238 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eb90, 0xc0011a2480, 0xc00102eb80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2239 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ede) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019696b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019696b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ec10, 0xc0011a2480, 0xc00102ec00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2240 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eec) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019696e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019696e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ec60, 0xc0011a2480, 0xc00102ec20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2241 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ec80, 0xc0011a2480, 0xc00102ec70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2242 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3edc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ecf0, 0xc0011a2480, 0xc00102ece0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2243 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3efd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019697d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019697d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ed20, 0xc0011a2480, 0xc00102ed10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2244 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ed40, 0xc0011a2480, 0xc00102ed30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2245 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ed60, 0xc0011a2480, 0xc00102ed50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2246 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eef) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ed80, 0xc0011a2480, 0xc00102ed70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2247 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eed) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eda0, 0xc0011a2480, 0xc00102ed90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2248 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f02) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102edc0, 0xc0011a2480, 0xc00102edb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2249 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f06) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ee10, 0xc0011a2480, 0xc00102ee00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2250 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0010303c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0010303c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ee30, 0xc0011a2480, 0xc00102ee20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2251 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0010303f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0010303f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ee50, 0xc0011a2480, 0xc00102ee40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2252 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ee70, 0xc0011a2480, 0xc00102ee60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2253 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f14) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ee90, 0xc0011a2480, 0xc00102ee80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2254 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f03) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eeb0, 0xc0011a2480, 0xc00102eea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2255 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f00) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0010304b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0010304b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eed0, 0xc0011a2480, 0xc00102eec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2256 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eef0, 0xc0011a2480, 0xc00102eee0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2257 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f04) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001030630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001030630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ef10, 0xc0011a2480, 0xc00102ef00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2258 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc000dec2a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000dec2a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ef50, 0xc0011a2480, 0xc00102ef20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2259 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001a56000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001a56000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ef70, 0xc0011a2480, 0xc00102ef60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2260 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ef90, 0xc0011a2480, 0xc00102ef80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2261 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102efb0, 0xc0011a2480, 0xc00102efa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2262 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f10) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102efd0, 0xc0011a2480, 0xc00102efc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2263 [semacquire]: sync.runtime_SemacquireMutex(0xc0075c9094, 0x489e100) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc0075c9090) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tick(0xc0075c9000, 0xc005016270, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:852 +0x10b github.com/cockroachdb/cockroach/pkg/storage.(*Store).processTick(0xc001f42700, 0x489e1a0, 0xc0019680c0, 0xbb, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3715 +0xf3 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019680c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:205 +0x290 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019680c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102eff0, 0xc0011a2480, 0xc00102efe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2264 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019680f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019680f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f020, 0xc0011a2480, 0xc00102f010) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2265 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eeb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f040, 0xc0011a2480, 0xc00102f030) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2266 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f1b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f060, 0xc0011a2480, 0xc00102f050) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2267 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f0a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f080, 0xc0011a2480, 0xc00102f070) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2268 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f09) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019681b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019681b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f0a0, 0xc0011a2480, 0xc00102f090) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2269 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3efa) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019681e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019681e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f0c0, 0xc0011a2480, 0xc00102f0b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2270 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f07) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f0e0, 0xc0011a2480, 0xc00102f0d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2271 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f12) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f100, 0xc0011a2480, 0xc00102f0f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2272 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f1a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f120, 0xc0011a2480, 0xc00102f110) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2273 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019682a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019682a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f140, 0xc0011a2480, 0xc00102f130) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2274 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f0c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019682d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019682d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f160, 0xc0011a2480, 0xc00102f150) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2275 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f15) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f180, 0xc0011a2480, 0xc00102f170) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2276 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f05) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f1a0, 0xc0011a2480, 0xc00102f190) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2277 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f24) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f1c0, 0xc0011a2480, 0xc00102f1b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2278 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f11) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019683f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019683f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f1e0, 0xc0011a2480, 0xc00102f1d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2279 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f19) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f200, 0xc0011a2480, 0xc00102f1f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2280 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f0e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f220, 0xc0011a2480, 0xc00102f210) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2281 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f1e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f240, 0xc0011a2480, 0xc00102f230) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2282 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019684b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019684b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f260, 0xc0011a2480, 0xc00102f250) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2283 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f17) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019684e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019684e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f290, 0xc0011a2480, 0xc00102f270) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2284 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f1c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f2c0, 0xc0011a2480, 0xc00102f2a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2285 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eab) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f2e0, 0xc0011a2480, 0xc00102f2d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2286 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f0d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f300, 0xc0011a2480, 0xc00102f2f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2287 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f16) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019685a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019685a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f340, 0xc0011a2480, 0xc00102f330) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2288 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f0b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019685d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019685d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f360, 0xc0011a2480, 0xc00102f350) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2289 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f1f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f380, 0xc0011a2480, 0xc00102f370) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2290 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eae) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f3a0, 0xc0011a2480, 0xc00102f390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2291 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ead) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f3c0, 0xc0011a2480, 0xc00102f3b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2292 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f18) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019686c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019686c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f3e0, 0xc0011a2480, 0xc00102f3d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2293 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f13) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019686f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019686f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f400, 0xc0011a2480, 0xc00102f3f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2294 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f0f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f420, 0xc0011a2480, 0xc00102f410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2295 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f20) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f440, 0xc0011a2480, 0xc00102f430) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2296 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f21) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f460, 0xc0011a2480, 0xc00102f450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2297 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019687b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019687b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f480, 0xc0011a2480, 0xc00102f470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2298 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f1d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019687e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019687e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f4a0, 0xc0011a2480, 0xc00102f490) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2299 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eac) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f4e0, 0xc0011a2480, 0xc00102f4b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2300 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f25) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f500, 0xc0011a2480, 0xc00102f4f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2301 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f29) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f550, 0xc0011a2480, 0xc00102f530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2302 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019688a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019688a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f570, 0xc0011a2480, 0xc00102f560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2303 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019688d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019688d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f590, 0xc0011a2480, 0xc00102f580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2304 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f26) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f5b0, 0xc0011a2480, 0xc00102f5a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2305 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f5d0, 0xc0011a2480, 0xc00102f5c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2306 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f5f0, 0xc0011a2480, 0xc00102f5e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2307 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f22) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019689c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019689c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f610, 0xc0011a2480, 0xc00102f600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2308 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f27) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019689f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019689f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f630, 0xc0011a2480, 0xc00102f620) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2309 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eff) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f650, 0xc0011a2480, 0xc00102f640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2310 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968a50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968a50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f670, 0xc0011a2480, 0xc00102f660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2311 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f6a0, 0xc0011a2480, 0xc00102f690) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2312 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f23) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968ab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968ab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f6c0, 0xc0011a2480, 0xc00102f6b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2313 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f08) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968ae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968ae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f6e0, 0xc0011a2480, 0xc00102f6d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2314 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102f710, 0xc0011a2480, 0xc00102f6f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2315 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968b40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968b40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fb00, 0xc0011a2480, 0xc00102f720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2316 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ecb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968b70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968b70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fb20, 0xc0011a2480, 0xc00102fb10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2317 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fb40, 0xc0011a2480, 0xc00102fb30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2318 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fb60, 0xc0011a2480, 0xc00102fb50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2319 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eba) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fb80, 0xc0011a2480, 0xc00102fb70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2320 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f28) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fba0, 0xc0011a2480, 0xc00102fb90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2321 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eaf) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fbc0, 0xc0011a2480, 0xc00102fbb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2322 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968cf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968cf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fbe0, 0xc0011a2480, 0xc00102fbd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2323 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fc00, 0xc0011a2480, 0xc00102fbf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2324 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eb1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fc20, 0xc0011a2480, 0xc00102fc10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2325 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ed2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fc40, 0xc0011a2480, 0xc00102fc30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2326 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ebd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fc60, 0xc0011a2480, 0xc00102fc50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2327 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fc80, 0xc0011a2480, 0xc00102fc70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2328 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ebf) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968e10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968e10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fca0, 0xc0011a2480, 0xc00102fc90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2329 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fcc0, 0xc0011a2480, 0xc00102fcb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2330 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ebe) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968e70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968e70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fce0, 0xc0011a2480, 0xc00102fcd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2331 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3eda) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968ea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968ea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fd10, 0xc0011a2480, 0xc00102fd00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2332 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ecc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fd30, 0xc0011a2480, 0xc00102fd20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2333 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ec4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fd50, 0xc0011a2480, 0xc00102fd40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2334 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ebc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019690b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019690b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fd70, 0xc0011a2480, 0xc00102fd60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2335 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ebb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001968a20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001968a20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fda0, 0xc0011a2480, 0xc00102fd90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2336 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fdc0, 0xc0011a2480, 0xc00102fdb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2337 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3edf) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fde0, 0xc0011a2480, 0xc00102fdd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2338 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3efb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fe00, 0xc0011a2480, 0xc00102fdf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2339 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ef0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019699b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019699b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fe20, 0xc0011a2480, 0xc00102fe10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2340 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3ee5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc0019699e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0019699e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fe40, 0xc0011a2480, 0xc00102fe30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2341 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3efc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fe60, 0xc0011a2480, 0xc00102fe50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2342 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3efe) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102fe80, 0xc0011a2480, 0xc00102fe70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2343 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001414550, 0x3f01) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001414540) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc002dc4100, 0x489e1a0, 0xc001969a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001969a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102feb0, 0xc0011a2480, 0xc00102fea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2344 [semacquire]: sync.runtime_Semacquire(0xc002dc4178) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc002dc4170) /usr/local/go/src/sync/waitgroup.go:130 +0x65 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ff00, 0xc0011a2480, 0xc00102fee0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2345 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc001f42700, 0x489e1a0, 0xc001969ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3769 +0x264 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ff60, 0xc0011a2480, 0xc00102ff10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2346 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc001f42700, 0x489e1a0, 0xc001969b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3825 +0x129 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00102ffe0, 0xc0011a2480, 0xc00102ff70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2347 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x489e1a0, 0xc001969b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1417 +0xe6 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217570, 0xc0011a2480, 0xc002d62720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2348 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc001969c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217580, 0xc0011a2480, 0xc002e54870) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2349 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc001969c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217590, 0xc0011a2480, 0xc002e548c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2350 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc001969cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002175a0, 0xc0011a2480, 0xc002e54910) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2352 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x489e1a0, 0xc001030030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x2ea github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217600, 0xc0011a2480, 0xc0002175e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2353 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc002e9a250, 0x9f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc002e9a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0007b0310, 0x489e1a0, 0xc0016c7410, 0xc000113c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x489e1a0, 0xc0016c7410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1628 +0x57 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc0016c7410, 0xc0016f8330, 0x25, 0x0, 0x0, 0xc002d62740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2354 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x489e1a0, 0xc001030090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1636 +0x2db github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217610, 0xc0011a2480, 0xc002d62760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2355 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x489e1a0, 0xc0010300c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:182 +0x181 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217630, 0xc0011a2480, 0xc002d62780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2356 [select]: github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x489e1a0, 0xc0010300f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:123 +0x148 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000217640, 0xc0011a2480, 0xc002d627a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2357 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x489e1a0, 0xc001030210) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00023a940, 0xc0011a2480, 0xc002d62920) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2358 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x489e1a0, 0xc001969fb0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00023abe0, 0xc0011a2480, 0xc002d629a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2359 [select]: github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc00114e780, 0xc00017f420, 0xc000354130, 0x40e5438, 0xc001164d20, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1014 +0x18b github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x489e1a0, 0xc001030510) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354140, 0xc0011a2480, 0xc0016c7bc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2360 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x489e1a0, 0xc001030570) /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x107 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003541f0, 0xc0011a2480, 0xc002d62a20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2361 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x489e1a0, 0xc001030600) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2024 +0x1fc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354330, 0xc0011a2480, 0xc0016c7c20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2362 [select]: github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x489e1a0, 0xc0010304e0) /go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x112 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354350, 0xc0011a2480, 0xc000354340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2363 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x489e1a0, 0xc001030690) /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1991 +0x80e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354370, 0xc0011a2480, 0xc002d62ae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2364 [select]: github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x489e1a0, 0xc001030660) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:147 +0x14f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003543d0, 0xc0011a2480, 0xc0003543b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2365 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x489e1a0, 0xc000dec1b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:324 +0xfa github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003543e0, 0xc0011a2480, 0xc002d62b40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2366 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x489e1a0, 0xc000dec270) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:682 +0x234 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003543f0, 0xc0011a2480, 0xc002d62b60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2367 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x489e1a0, 0xc000dec330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:488 +0x479 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354580, 0xc0011a2480, 0xc002e54a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2402 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x489e1a0, 0xc0010ad080) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354c90, 0xc0011a2480, 0xc00191a4c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1883 [IO wait]: internal/poll.runtime_pollWait(0x7f086d319db8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001ae2398, 0x72, 0x700, 0x719, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001ae2380, 0xc002e7d000, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001ae2380, 0xc002e7d000, 0x719, 0x719, 0x203001, 0x48314e0, 0x72b4ea0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0025e41c0, 0xc002e7d000, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0072a3dc0, 0xc002e7d000, 0x719, 0x719, 0x0, 0x48317a0, 0xc00469b618) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001078258, 0x482c340, 0xc0072a3dc0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001078000, 0x48317e0, 0xc0025e41c0, 0x5, 0xc0025e41c0, 0xc003100400) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001078000, 0x40e5100, 0xc001078138, 0xed500a8d3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001078000, 0xc002413000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc001761820, 0xc002413000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1896 +0x110 bufio.(*Reader).fill(0xc001a51248) /usr/local/go/src/bufio/bufio.go:100 +0x10f bufio.(*Reader).ReadByte(0xc001a51248, 0x1, 0x1, 0x2) /usr/local/go/src/bufio/bufio.go:249 +0x39 github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc001a519a0, 0x486e520, 0xc001a51248, 0x4860120, 0x72b56a8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc001a51200, 0x489e0e0, 0xc001570900, 0xc001811590, 0xc0018fe000, 0x5400, 0x15000, 0xc001636f18, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:313 +0x474 github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x489e0e0, 0xc001570900, 0x48f9e60, 0xc001078000, 0xc002e5701b, 0x4, 0xc0013ebb00, 0x485eca0, 0xc001115050, 0x4000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:168 +0x18d github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc001636d80, 0x489e0e0, 0xc001570900, 0x48f9e60, 0xc001078000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:536 +0x87e github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x48faa00, 0xc0025e41c0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1813 +0x165 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc0011a2480, 0x489e1a0, 0xc001114fc0, 0xc00043c3b8, 0x48faa00, 0xc0025e41c0, 0xc001114ff0) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xa9 created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b goroutine 2403 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x489e1a0, 0xc0010ad0b0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:297 +0x12b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354cc0, 0xc0011a2480, 0xc0010aca80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2404 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x489e1a0, 0xc0010ad0e0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:308 +0xda github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354ce0, 0xc0011a2480, 0xc00191a4e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2405 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x489e1a0, 0xc0010ad110) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:322 +0xfb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354d00, 0xc0011a2480, 0xc0010acb10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2406 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x489e1a0, 0xc0010ad050) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:246 +0x193 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000354d20, 0xc0011a2480, 0xc00191a500) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2408 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x489e1a0, 0xc0010adef0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:171 +0x536 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003554d0, 0xc0011a2480, 0xc00191abe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2410 [IO wait]: internal/poll.runtime_pollWait(0x7f086d31a778, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001fc6298, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001fc6280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001fc6280, 0xc0025e4101, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000bcc768, 0xc001617e68, 0xc001617e70, 0x38) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000bcc768, 0x40e2110, 0xc0011a2480, 0x489e1a0, 0xc001114fc0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc00043c3b8, 0x489e1a0, 0xc001114fc0, 0xc0011a2480, 0x4886160, 0xc000bcc768, 0xc001114ff0, 0xc001114fc0, 0xc0016c9768) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x489e1a0, 0xc001114fc0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1809 +0xdd github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000355560, 0xc0011a2480, 0xc0010ad830) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3921 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc003f12b90, 0xc003fa4270, 0x5, 0x5, 0xc005356038, 0xc002e04015, 0x21) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003f12b90, 0xc003fa4270, 0x5, 0x5, 0xc11b10, 0xc003e18660, 0xc00448b978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003f9ad20, 0xc003fa4270, 0x5, 0x5, 0xc00004e000, 0xc00448b9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003f9ad20, 0xc003fa4270, 0x5, 0x5, 0x5, 0xc00448ba50, 0xc372ac, 0xc005356000) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003944700, 0xc003fa4270, 0x5, 0x5, 0x83e57d, 0xc005356000, 0x4831580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003fa4260, 0x7fffffff, 0x72b56a8, 0xc0045aa6c0, 0x36, 0x600, 0x3de8800, 0xc005356000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003fa4260, 0xc003944700, 0x0, 0x0, 0x7fffffff, 0xc00448bcb8, 0x4886820, 0x72b56a8, 0x3e9a360, 0x485a160, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003fa4260, 0x7f086d36adf0, 0xc000284fd0, 0xc003944700, 0x0, 0x0, 0x3de8800, 0xc00531d1c0, 0x7fffffff, 0xc00448bcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc003f04800, 0x3de8800, 0xc00531d1c0, 0xc00448bcb8, 0xc007643860, 0xc00531d1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc003f04800, 0xc000089970, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc003af0ea0, 0xc00448be00, 0xc00448bdf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc003af0ea0, 0x3de8800, 0xc00531d1c0, 0x2c5e8e0, 0xc000c91500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc003acc960, 0x489e1a0, 0xc003f9adb0, 0xc005356000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc003acc960, 0xc003f06500, 0xc0010e50e0, 0x489e1a0, 0xc003f9adb0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003f9adb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003acc980, 0xc00108b320, 0xc003f9ad50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2413 [select]: database/sql.(*DB).connectionOpener(0xc001268780, 0x489e0e0, 0xc000e4e780) /usr/local/go/src/database/sql/sql.go:1000 +0xe8 created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:670 +0x15e goroutine 2414 [select]: database/sql.(*DB).connectionResetter(0xc001268780, 0x489e0e0, 0xc000e4e780) /usr/local/go/src/database/sql/sql.go:1013 +0xfb created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:671 +0x194 goroutine 2492 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0013d8400, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002454cc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0024ca480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2493 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0024ca480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2607 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214960, 0x48fa580, 0xc00111f850, 0x485c120, 0xc000214960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc001df2d80, 0x0, 0xc0029111a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc001df2d80, 0xc0003e9c20, 0x40e1ba8, 0x3c89ac0, 0x47cf94f) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc001df2d80, 0xc0003e9c20, 0x40e1ba8, 0x489e1a0, 0xc0011c2930) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc002cb4480, 0xc0018d5c00, 0xc000c792f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc002cb4480, 0xc0018d5c00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00030b160, 0xc0025ed980, 0x48fef60, 0xc002cb4480, 0xc0018d5c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2586 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0035a0000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2461 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0018b6e80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2221 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0026d7080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 10144 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0032be8c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2462 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000cf1040, 0xc001745860, 0xc0018964b0, 0xf, 0xc00140b2c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000682e40) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00172fb50, 0xc000cfe090, 0xc00114e460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6781 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0035a01c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1855 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0010337c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001fa2240, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc002cb4480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 241325 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc0010a4c00, 0x0, 0x1, 0xc0021cc180, 0xc000f47618) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0x6, 0xc001832fc0, 0xc003b15000, 0x15c1014bc7fc9752) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0002fc980, 0x33, 0x0, 0x0, 0xc0063ded20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2654 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214870, 0x48fa580, 0xc001a9d7f0, 0x485c120, 0xc000214870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc00190a9c0, 0x0, 0xc002ca4c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc00190a9c0, 0xc002c68ec0, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc00190a9c0, 0xc002c68ec0, 0x40e1ba8, 0x489e1a0, 0xc00187fa70) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc002c43200, 0xc000177900, 0xc000aa7500, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc002c43200, 0xc000177900, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001993d10, 0xc000ddc180, 0x48fef60, 0xc002c43200, 0xc000177900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2433 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000c9a090, 0xa1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000c9a080) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc000839260, 0x489e1a0, 0xc001108b40, 0xc0029b64e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc001108b40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e1a0, 0xc001108b40, 0xc000232080, 0x1d, 0x0, 0x0, 0xc0002c4440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1856 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002cb4480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 6363 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc004074340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 6364 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0016ad380, 0xc004052180, 0xc0002f4820, 0xf, 0xc003eda420, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc004544570) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003c821c0, 0xc00108b320, 0xc003f122d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3976 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc003e34c80, 0xc003e7cbf0, 0x5, 0x5, 0x0, 0x0, 0xc0063f1200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003e34c80, 0xc003e7cbf0, 0x5, 0x5, 0xc11b10, 0xc003bd87e0, 0xc001edbaf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003e86f30, 0xc003e7cbf0, 0x5, 0x5, 0x38, 0xc001edbb20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003e86f30, 0xc003e7cbf0, 0x5, 0x5, 0x5, 0xc001edbbd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003bbae00, 0xc003e7cbf0, 0x5, 0x5, 0x203002, 0x203002, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003e7cbe0, 0x7fffffff, 0x732a2f, 0xc000b8c900, 0x4, 0xc001edbc70, 0x733ecb, 0xc000c3aa80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003e7cbe0, 0xc003bbae00, 0x0, 0x0, 0x7fffffff, 0xc001edbe30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003e7cbe0, 0x7f086d36adf0, 0xc000284fd0, 0xc003bbae00, 0x0, 0x0, 0x3de86e0, 0xc0081706e0, 0x7fffffff, 0xc001edbe30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0039a3080, 0x3de86e0, 0xc0081706e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc003993780, 0x489e1a0, 0xc003e86ff0, 0xc0063f1200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc003993780, 0xc000214870, 0x489e1a0, 0xc003e86ff0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003e86ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0039937a0, 0xc000cfe090, 0xc003e86fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2587 [IO wait]: internal/poll.runtime_pollWait(0x7f086d319ce8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00007d798, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00007d780, 0xc002df7800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00007d780, 0xc002df7800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0025e4450, 0xc002df7800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc007afb9e0, 0xc002df7800, 0x791, 0x791, 0x12a, 0x482c600, 0xc00177da78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0017cb758, 0x482c340, 0xc007afb9e0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0017cb500, 0x48317e0, 0xc0025e4450, 0x5, 0xc0025e4450, 0xc007890060) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0017cb500, 0x40e5100, 0xc0017cb638, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0017cb500, 0xc003580000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0027a1aa0, 0xc0033ea3b8, 0x9, 0x9, 0xc006e16dd8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0027a1aa0, 0xc0033ea3b8, 0x9, 0x9, 0x9, 0xc006b322d0, 0x485e320, 0xc007199620) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0033ea3b8, 0x9, 0x9, 0x482c140, 0xc0027a1aa0, 0x0, 0xc000000000, 0xc00323d700, 0x7890000) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0033ea380, 0xc007890030, 0xc007890030, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0035a0000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 9887 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0034e8840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 244104 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xba, 0x490bae0, 0xc00689c060, 0x0, 0x1, 0xc000be0b68, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xba, 0xc006886240, 0xc003f5f800, 0x15c1014bdc1f46c4) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0060ff800, 0x33, 0x0, 0x0, 0xc006661a10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1857 [IO wait]: internal/poll.runtime_pollWait(0x7f086d319c18, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00007d818, 0x72, 0x5900, 0x599e, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00007d800, 0xc003dfc000, 0x599e, 0x599e, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00007d800, 0xc003dfc000, 0x599e, 0x599e, 0xc0034c1938, 0x5901, 0xc0016d3a00) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0001103e8, 0xc003dfc000, 0x599e, 0x599e, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002d48840, 0xc003dfc000, 0x599e, 0x599e, 0xc003dfc00d, 0x5a, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0073800a0, 0xc003dfc000, 0x599e, 0x599e, 0x77, 0x482c600, 0xc0034c1a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0016d3ad8, 0x482c340, 0xc0073800a0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0016d3880, 0x7f086d36ae18, 0xc002d48840, 0x5, 0xc002d48840, 0x5a) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0016d3880, 0x40e5100, 0xc0016d39b8, 0x50) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0016d3880, 0xc0034f0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002911200, 0xc002c261f8, 0x9, 0x9, 0xc001a84288, 0xc00170fe00, 0x51) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002911200, 0xc002c261f8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc002c261f8, 0x9, 0x9, 0x482c140, 0xc002911200, 0x0, 0x700000000, 0xc0018d5c00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc002c261c0, 0xc0017d2990, 0xc0017d2990, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002cb4480, 0xc0017d2c00, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0025ed980, 0x48fef60, 0xc002cb4480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0025ed980, 0x48fef60, 0xc002cb4480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2608 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc00109ecd0, 0xc0003e9c10, 0x5, 0x5, 0x0, 0x0, 0xc00009ea00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00109ecd0, 0xc0003e9c10, 0x5, 0x5, 0xc11b10, 0xc0017475a0, 0xc005f63af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0011c28a0, 0xc0003e9c10, 0x5, 0x5, 0x3a, 0xc005f63b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0011c28a0, 0xc0003e9c10, 0x5, 0x5, 0x5, 0xc005f63bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0018d5c00, 0xc0003e9c10, 0x5, 0x5, 0x203001, 0x203001, 0x713a95) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003e9c00, 0x7fffffff, 0x732a2f, 0xc001abe000, 0x4, 0xc005f63c70, 0x733ecb, 0xc002804ba0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003e9c00, 0xc0018d5c00, 0x0, 0x0, 0x7fffffff, 0xc005f63e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003e9c00, 0x7f086d36adf0, 0xc000284fd0, 0xc0018d5c00, 0x0, 0x0, 0x3de86e0, 0xc007380100, 0x7fffffff, 0xc005f63e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001df2d80, 0x3de86e0, 0xc007380100, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc00111f850, 0x489e1a0, 0xc0011c2990, 0xc0063f0400) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc00111f850, 0xc000214960, 0x489e1a0, 0xc0011c2990, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0011c2990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00111f870, 0xc0011a2480, 0xc0011c2960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2651 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214960, 0x1, 0xc00360e180, 0xc001171900, 0x48fa520, 0xc001a9d720, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc00187f560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9d680, 0xc0011a2480, 0xc00187f500) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3919 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0010e50e0, 0x1, 0xc003e13320, 0xc003f06500, 0x48fa520, 0xc003acc960, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003f9ab40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003acc8f0, 0xc00108b320, 0xc003f9ab10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1690 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0017ca000, 0xc00357b560, 0x489e1a0, 0xc0017d3920) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2698 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453f40, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00140ff98, 0x72, 0x8400, 0x845e, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00140ff80, 0xc003928000, 0x845e, 0x845e, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00140ff80, 0xc003928000, 0x845e, 0x845e, 0xc003675938, 0x8401, 0xc000feac00) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000bcc5e0, 0xc003928000, 0x845e, 0x845e, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00189b3e0, 0xc003928000, 0x845e, 0x845e, 0xc00392800d, 0x60, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc006320e00, 0xc003928000, 0x845e, 0x845e, 0x7d, 0x482c600, 0xc003675a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000feacd8, 0x482c340, 0xc006320e00, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000feaa80, 0x7f086d36ae18, 0xc00189b3e0, 0x5, 0xc00189b3e0, 0x60) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000feaa80, 0x40e5100, 0xc000feabb8, 0x7) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000feaa80, 0xc003666000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001752ea0, 0xc000218e38, 0x9, 0x9, 0xc001415108, 0xc005f4a600, 0x57) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001752ea0, 0xc000218e38, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000218e38, 0x9, 0x9, 0x482c140, 0xc001752ea0, 0x0, 0x700000000, 0xc004b8cf00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000218e00, 0xc001250120, 0xc001250120, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0032ca300, 0xc001250570, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0025ed980, 0x48fef60, 0xc0032ca300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0025ed980, 0x48fef60, 0xc0032ca300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 242768 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).addToPurgatoryLocked.func2(0x489e1a0, 0xc0015aa3f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:1086 +0x325 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0011f5850, 0xc00108b320, 0xc001a23920) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2953 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000358e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 3920 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc00176b180, 0xc003af0ea0, 0x489e1a0, 0xc003f9ac00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2697 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0032ca300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2882 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001d1d8c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2676 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0035f29b0, 0xc0018a42f0, 0x5, 0x5, 0xbe, 0xc0016719b8, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0035f29b0, 0xc0018a42f0, 0x5, 0x5, 0xc11b10, 0xc0018988a0, 0xc002dc9858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0017d3a40, 0xc0018a42f0, 0x5, 0x5, 0xc0040372c0, 0xc002dc9880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0017d3a40, 0xc0018a42f0, 0x5, 0x5, 0x5, 0xc0040372c0, 0x20, 0x18) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0015a8b00, 0xc0018a42f0, 0x5, 0x5, 0xc004cc0140, 0x40, 0xc00018dc00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0018a42e0, 0x7fffffff, 0xc0e586, 0xc0034ee450, 0x58, 0x58, 0xc002316c60, 0xc0013e8c80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0018a42e0, 0xc0015a8b00, 0x0, 0x0, 0x7fffffff, 0xc002dc9b98, 0x4886820, 0x72b56a8, 0x40, 0xc006c93415, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0018a42e0, 0x7f086d36adf0, 0xc000284fd0, 0xc0015a8b00, 0x0, 0x0, 0x3de9580, 0xc00344f2c0, 0x7fffffff, 0xc002dc9b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001503900, 0x3de9580, 0xc00344f2c0, 0xc002dc9b98, 0x0, 0xc00344f2c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001503900, 0x5, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00357b560, 0xc002dc9ce0, 0xc002dc9cd0, 0xc002a0cb40, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00357b560, 0x3de9580, 0xc00344f2c0, 0xc004037280, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc000423740, 0xc004037280, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc0017d3980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013e8030, 0xc000cfe090, 0xc002eda060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2677 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc0017cc3c0, 0x7f086677d630, 0xc0013e80e0, 0x7f086677d630, 0xc0013e80e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc0017cc3c0, 0x48fa760, 0xc0013e80e0, 0xc02f00010301, 0xc0013e80e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0003e93e0, 0x48eb940, 0xc00206c780, 0x0, 0xc0029111a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0003e93e0, 0x48eb940, 0xc00206c780, 0xc002eda1e0, 0x40e1e60, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0003e93e0, 0x48eb940, 0xc00206c780, 0xc002eda1e0, 0x40e1e60, 0x489e1a0, 0xc00184e600) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc002cb4480, 0xc0029ea000, 0xc000ffc5a0, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc002cb4480, 0xc0029ea000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00030b160, 0xc0025ed980, 0x48fef60, 0xc002cb4480, 0xc0029ea000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2678 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc002e9a250, 0xa1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc002e9a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0007b0310, 0x489e1a0, 0xc00184e600, 0xc0027a0060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc00184e600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc00184e600, 0xc00323e040, 0x1d, 0x0, 0x0, 0xc002eda220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7334 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001290700, 0xc003bcaa20, 0x489e1a0, 0xc003423230) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2835 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc000b58960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013f8120, 0xc00108b320, 0xc000356460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2822 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc0017cc3c0, 0x7f086677d630, 0xc0013e8ad0, 0x7f086677d630, 0xc0013e8ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc0017cc3c0, 0x48fa760, 0xc0013e8ad0, 0xc02f00010301, 0xc0013e8ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0003e93e0, 0x48eb940, 0xc00206d140, 0x0, 0xc001752de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0003e93e0, 0x48eb940, 0xc00206d140, 0xc000390a80, 0x40e1e60, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0003e93e0, 0x48eb940, 0xc00206d140, 0xc000390a80, 0x40e1e60, 0x489e1a0, 0xc0011eff80) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc0029eb400, 0xc000ffc5a0, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc0029eb400, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003f74d0, 0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc0029eb400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2886 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002e58380, 0xc001434c60, 0x489e1a0, 0xc001114720) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2823 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc002e9a250, 0xa0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc002e9a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0007b0310, 0x489e1a0, 0xc0011eff80, 0xc000c3b800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc0011eff80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc0011eff80, 0xc00323e900, 0x1d, 0x0, 0x0, 0xc000390ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2881 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000cf1040, 0xc00228ac00, 0xc0018964b0, 0xf, 0xc0029e9320, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0010f5fb0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001159010, 0xc000cfe090, 0xc000c73860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6964 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00419f500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2880 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0012527c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2888 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0011ac380, 0xc001434ea0, 0x489e1a0, 0xc001114a50) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 6868 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00399e380, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003d3a780, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003981500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2889 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc001114db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001159350, 0xc0011a2480, 0xc0017611c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2890 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc000327080, 0x7f086677d630, 0xc00031d490, 0x7f086677d630, 0xc00031d490) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc000327080, 0x48fa760, 0xc00031d490, 0xc02f00010301, 0xc00031d490) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0002ba420, 0x48eb940, 0xc002dd6240, 0x0, 0xc002ca4c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0002ba420, 0x48eb940, 0xc002dd6240, 0xc0002c4520, 0x40e1e60, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0002ba420, 0x48eb940, 0xc002dd6240, 0xc0002c4520, 0x40e1e60, 0x489e1a0, 0xc001108d50) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc002c43200, 0xc003786600, 0xc000de86f0, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc002c43200, 0xc003786600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001993d10, 0xc000ddc180, 0x48fef60, 0xc002c43200, 0xc003786600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 243182 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152b340, 0x489e120, 0xc00010c010, 0xbb, 0x490bae0, 0xc006b4afc0, 0x0, 0x1, 0x3e8e720, 0x6) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c3cea0, 0x489e120, 0xc00010c010, 0xbb, 0xc0067879e0, 0xc0051d4000, 0x15c1014bdedff3b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc0039a4a40, 0x33, 0x0, 0x0, 0xc006a4e4b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2957 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0029b8aa0, 0xc0017dc630, 0x5, 0x5, 0xc0065a0000, 0x1f0, 0xc0080a4400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0029b8aa0, 0xc0017dc630, 0x5, 0x5, 0xc11b10, 0xc00121dac0, 0xc0038c5af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0015c72f0, 0xc0017dc630, 0x5, 0x5, 0x70, 0xc0038c5b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0015c72f0, 0xc0017dc630, 0x5, 0x5, 0x5, 0xc0038c5bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002dce600, 0xc0017dc630, 0x5, 0x5, 0x203001, 0x203001, 0x7f086674b720) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0017dc620, 0x7fffffff, 0x30, 0xc00759f350, 0x4, 0xc0038c5c70, 0xc001a70000, 0x7f086d4986d0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0017dc620, 0xc002dce600, 0x0, 0x0, 0x7fffffff, 0xc0038c5e30, 0x4886820, 0x72b56a8, 0x8, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0017dc620, 0x7f086d36adf0, 0xc000284fd0, 0xc002dce600, 0x0, 0x0, 0x3de86e0, 0xc001f65000, 0x7fffffff, 0xc0038c5e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001df2540, 0x3de86e0, 0xc001f65000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc001a9c020, 0x489e1a0, 0xc0015c73b0, 0xc0080a45f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc001a9c020, 0xc000214960, 0x489e1a0, 0xc0015c73b0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0015c73b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9c040, 0xc0011a2480, 0xc0015c7380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2956 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214960, 0x48fa580, 0xc001a9c020, 0x485c120, 0xc000214960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc001df2540, 0x0, 0xc002a6fc20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc001df2540, 0xc0017dc680, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc001df2540, 0xc0017dc680, 0x40e1ba8, 0x489e1a0, 0xc0015c7350) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc001683e00, 0xc002dce600, 0xc000c792f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc001683e00, 0xc002dce600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000323b00, 0xc0025ed980, 0x48fef60, 0xc001683e00, 0xc002dce600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2955 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0018b6b00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001924780, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000358e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 5723 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0016ad380, 0xc00282ed80, 0xc000368270, 0xf, 0xc003edbb60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0042363f0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00198cd20, 0xc00108b320, 0xc003d52a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 10183 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003a6bb80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0023205a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003c2a000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 3922 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214870, 0x48fa580, 0xc003acca30, 0x485c120, 0xc000214870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc0018e2cc0, 0x0, 0xc000c2a3c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc0018e2cc0, 0xc003fa4460, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc0018e2cc0, 0xc003fa4460, 0x40e1ba8, 0x489e1a0, 0xc003f9af90) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc00149db00, 0xc003944800, 0xc000aa7500, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc00149db00, 0xc003944800, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003e5a90, 0xc000ddc180, 0x48fef60, 0xc00149db00, 0xc003944800) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 6869 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003981500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 242721 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot(0xc000214960, 0x48fa640, 0xc0062a9450, 0x485c120, 0xc000214960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:420 +0x1cf github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftSnapshot_Handler(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc006c49a40, 0x0, 0xc001752de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:717 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc006c49a40, 0xc00718a520, 0x40e1bb0, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc006c49a40, 0xc00718a520, 0x40e1bb0, 0x489e1a0, 0xc0070f8f30) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc00323d600, 0xc000c792f0, 0x6fa4a00, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc00323d600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003f74d0, 0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc00323d600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 3975 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214870, 0x48fa580, 0xc003993780, 0x485c120, 0xc000214870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc0039a3080, 0x0, 0xc002dc0540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc0039a3080, 0xc003e7cc00, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214870, 0x48eb940, 0xc0039a3080, 0xc003e7cc00, 0x40e1ba8, 0x489e1a0, 0xc003e86f90) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000ddc180, 0x48fef60, 0xc001abfc80, 0xc003bbae00, 0xc000aa7500, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000ddc180, 0x48fef60, 0xc001abfc80, 0xc003bbae00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00178e8f0, 0xc000ddc180, 0x48fef60, 0xc001abfc80, 0xc003bbae00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 3974 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc003e34aa0, 0xc003e7ca50, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003e34aa0, 0xc003e7ca50, 0x5, 0x5, 0xc11b10, 0xc003bd87c0, 0xc00449f978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003e86d20, 0xc003e7ca50, 0x5, 0x5, 0x74cff0, 0xc00449f9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003e86d20, 0xc003e7ca50, 0x5, 0x5, 0x5, 0xf, 0xc000e8fda0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003bbad00, 0xc003e7ca50, 0x5, 0x5, 0x83e500, 0xc0048b0ce0, 0xc00449fa78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003e7ca40, 0x7fffffff, 0x3c06120, 0xc000e8fda0, 0xc00449fa0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003e7ca40, 0xc003bbad00, 0x0, 0x0, 0x7fffffff, 0xc00449fcb8, 0x4886820, 0x72b56a8, 0x0, 0xc003b13560, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003e7ca40, 0x7f086d36adf0, 0xc000284fd0, 0xc003bbad00, 0x0, 0x0, 0x3de8800, 0xc00297e140, 0x7fffffff, 0xc00449fcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc003c12680, 0x3de8800, 0xc00297e140, 0xc00449fcb8, 0xc003e57e50, 0xc00297e140) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc003c12680, 0xc000082a70, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc003bcbd40, 0xc00449fe00, 0xc00449fdf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc003bcbd40, 0x3de8800, 0xc00297e140, 0x2c5e8e0, 0xc001f42700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0039936c0, 0x489e1a0, 0xc003e86db0, 0xc004af2d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0039936c0, 0xc0039a5a00, 0xc000214960, 0x489e1a0, 0xc003e86db0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003e86db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0039936e0, 0xc0011a2480, 0xc003e86d50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3973 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002c32700, 0xc003bcbd40, 0x489e1a0, 0xc003e86c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 3972 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214960, 0x1, 0xc003c0d860, 0xc0039a5a00, 0x48fa520, 0xc0039936c0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003e86b40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003993650, 0xc0011a2480, 0xc003e86b10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 4067 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc003f12dc0, 0xc003fa4450, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003f12dc0, 0xc003fa4450, 0x5, 0x5, 0xc11b10, 0xc003e18680, 0xc003683af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003f9af30, 0xc003fa4450, 0x5, 0x5, 0x0, 0xc003683b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003f9af30, 0xc003fa4450, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003944800, 0xc003fa4450, 0x5, 0x5, 0x0, 0x100000001, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003fa4440, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003fa4440, 0xc003944800, 0x0, 0x0, 0x7fffffff, 0xc003683e30, 0x4886820, 0x72b56a8, 0x8, 0x706f17, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003fa4440, 0x7f086d36adf0, 0xc000284fd0, 0xc003944800, 0x0, 0x0, 0x3de86e0, 0xc0075e77e0, 0x7fffffff, 0xc003683e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0018e2cc0, 0x3de86e0, 0xc0075e77e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc003acca30, 0x489e1a0, 0xc003f9aff0, 0xc00602a800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc003acca30, 0xc000214870, 0x489e1a0, 0xc003f9aff0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003f9aff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003acca50, 0xc000cfe090, 0xc003f9afc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6300 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002d5cdc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 7336 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214960, 0x48fa580, 0xc0028b1c50, 0x485c120, 0xc000214960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc00206ca80, 0x0, 0xc001752de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc00206ca80, 0xc0031eea60, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc00206ca80, 0xc0031eea60, 0x40e1ba8, 0x489e1a0, 0xc003423650) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc004b8cf00, 0xc000c792f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc004b8cf00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003f74d0, 0xc0025ed980, 0x48fef60, 0xc0032ca300, 0xc004b8cf00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 7339 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000359dc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 244528 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc006f4c300, 0x0, 0x1, 0x2c522bd, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0x6, 0xc0066be5a0, 0xc004b0b800, 0x15c1014be5a61bd8) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc00027d8c0, 0x33, 0x0, 0x0, 0xc006ae0960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 241601 [IO wait]: internal/poll.runtime_pollWait(0x7f086d44d078, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001584a18, 0x72, 0x4100, 0x4114, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001584a00, 0xc003183000, 0x4114, 0x4114, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001584a00, 0xc003183000, 0x4114, 0x4114, 0x203001, 0x18e00a5c1c716a28, 0x942a958fd0bbea43) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000010020, 0xc003183000, 0x4114, 0x4114, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0047d2b00, 0xc003183000, 0x4114, 0x4114, 0x380, 0x373, 0xc0044350e8) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00180f3d8, 0x482c340, 0xc0047d2b00, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00180f180, 0x48317e0, 0xc000010020, 0x5, 0xc000010020, 0xc000097900) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00180f180, 0x40e5100, 0xc00180f2b8, 0xd9) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00180f180, 0xc0006ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000cc5020, 0xc001120860, 0x5, 0x200, 0x40e5218, 0xc004435420, 0x18) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000cc5020, 0xc001120860, 0x5, 0x200, 0x5, 0xbc, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recvMessage(0xc001120840, 0xc004435580, 0xbc, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:929 +0x104 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1Buf(0xc001120840, 0xc004435580, 0xb7) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:978 +0x39 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:999 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).simpleExec(0xc001120840, 0xc0005dacc0, 0xb6, 0xc004435668, 0x0, 0x736ebc, 0x0, 0x3aaaa20, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:603 +0x1ee github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).Exec(0xc001120840, 0xc0005dacc0, 0xb6, 0x72b56a8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:857 +0x247 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).ExecContext(0xc001120840, 0x489e120, 0xc00010c010, 0xc0005dacc0, 0xb6, 0x72b56a8, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:42 +0x195 database/sql.ctxDriverExec(0x489e120, 0xc00010c010, 0x7f08667be300, 0xc001120840, 0x0, 0x0, 0xc0005dacc0, 0xb6, 0x72b56a8, 0x0, ...) /usr/local/go/src/database/sql/ctxutil.go:31 +0x233 database/sql.(*DB).execDC.func2() /usr/local/go/src/database/sql/sql.go:1467 +0x1db database/sql.withLock(0x4859960, 0xc001ae2a00, 0xc0044359b8) /usr/local/go/src/database/sql/sql.go:3097 +0x63 database/sql.(*DB).execDC(0xc0011b8180, 0x489e120, 0xc00010c010, 0xc001ae2a00, 0xc004435ab8, 0xc0005dacc0, 0xb6, 0x0, 0x0, 0x0, ...) /usr/local/go/src/database/sql/sql.go:1462 +0x47d database/sql.(*DB).exec(0xc0011b8180, 0x489e120, 0xc00010c010, 0xc0005dacc0, 0xb6, 0x0, 0x0, 0x0, 0x40e5101, 0x33, ...) /usr/local/go/src/database/sql/sql.go:1447 +0x139 database/sql.(*DB).ExecContext(0xc0011b8180, 0x489e120, 0xc00010c010, 0xc0005dacc0, 0xb6, 0x0, 0x0, 0x0, 0x485f9e0, 0xc0047d2ae0, ...) /usr/local/go/src/database/sql/sql.go:1425 +0xde github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Exec(0xc0013f8670, 0x49240c0, 0xc004fa9100, 0xc0005dacc0, 0xb6, 0x0, 0x0, 0x0, 0x485f9e0, 0xc0047d2ae0) /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:52 +0xbc github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestRepartitioning.func1(0xc004fa9100) /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/partition_test.go:1299 +0x1f2 testing.tRunner(0xc004fa9100, 0xc0070720c0) /usr/local/go/src/testing/testing.go:865 +0xc0 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:916 +0x35a goroutine 241716 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xbe, 0x490bae0, 0xc0029895c0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xbe, 0xc003dc1d40, 0xc002a46f38, 0xc005bc32e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc00491f100, 0x33, 0x0, 0x0, 0xc003cf4300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 134409 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x489e1a0, 0xc0015c6a20) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:258 +0x11f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e1a0, 0xc0015c6a20, 0xc0062de000, 0x2a, 0x0, 0x0, 0xc007295320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10331 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0039aa910, 0xc003f61310, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0039aa910, 0xc003f61310, 0x5, 0x5, 0xc11b10, 0xc0002af2a0, 0xc0037df978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003ccb050, 0xc003f61310, 0x5, 0x5, 0x74cff0, 0xc0037df9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003ccb050, 0xc003f61310, 0x5, 0x5, 0x5, 0xf, 0xc000079c20, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003c7cd00, 0xc003f61310, 0x5, 0x5, 0x83e500, 0xc00684f320, 0xc0037dfa78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003f61300, 0x7fffffff, 0x3c06120, 0xc000079c20, 0xc0037dfa0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003f61300, 0xc003c7cd00, 0x0, 0x0, 0x7fffffff, 0xc0037dfcb8, 0x4886820, 0x72b56a8, 0x0, 0xc003e33400, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003f61300, 0x7f086d36adf0, 0xc000284fd0, 0xc003c7cd00, 0x0, 0x0, 0x3de8800, 0xc00554a880, 0x7fffffff, 0xc0037dfcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0043e8680, 0x3de8800, 0xc00554a880, 0xc0037dfcb8, 0xc0067d3f90, 0xc00554a880) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0043e8680, 0xc00009a070, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc002134360, 0xc0037dfe00, 0xc0037dfdf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc002134360, 0x3de8800, 0xc00554a880, 0x2c5e8e0, 0xc000c91500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc003c83330, 0x489e1a0, 0xc003ccb170, 0xc00173ffc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc003c83330, 0xc00390afc0, 0xc0010e50e0, 0x489e1a0, 0xc003ccb170, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003ccb170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003c83360, 0xc00108b320, 0xc003ccb080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 243084 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001935500, 0xc004b36be9, 0xc007c9b5f8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000d27a00, 0x3e99440, 0xc005942780, 0xc007cab650, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000d27a00, 0xc005082d80, 0x36) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc005140a20, 0xc007cab798, 0xc007cab788, 0xc0002dd260, 0xc004b36be9) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc005140a20, 0x3e99440, 0xc005942780, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e160, 0xc005942660, 0x3fc5646, 0x21, 0x3f4fce0, 0xc000d27900, 0x3e99440, 0xc005942780, 0xc000d2bc00, 0xc001481c40, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e160, 0xc005942660, 0x3fc5646, 0x21, 0x3f4fce0, 0xc000d27900, 0x3e99440, 0xc005942780, 0xc000d2bc00, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc000d2bc00, 0x489e160, 0xc005942660, 0x3fc5646, 0x21, 0x3f4fce0, 0xc000d27900, 0x3e99440, 0xc005942780, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc000bcce48, 0x489e160, 0xc005942660, 0xc000d27900, 0x0, 0x0, 0x0, 0x100000002, 0x489e160, 0xc005942660) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc003f03840, 0x489e160, 0xc005942660, 0x2, 0x485a560, 0xc000bcce48, 0x0, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc003f03840, 0x489e160, 0xc005942660, 0x0, 0x0, 0x200000002, 0x3, 0xc00039c96c, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000cd5c20, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000cd5c20, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000cd5c20, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000cd5c20, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000cd5c20, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000cd5c20, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc00007d660, 0x489e160, 0xc005942660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.lookupRangeFwdScan(0x489e160, 0xc005942660, 0x482c760, 0xc00007d660, 0xc004b36a50, 0x3, 0x8, 0x0, 0x0, 0x1dcd6500, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/range_lookup.go:328 +0x3f2 github.com/cockroachdb/cockroach/pkg/internal/client.RangeLookup(0x489e160, 0xc005942660, 0x482c760, 0xc00007d660, 0xc004b36a50, 0x3, 0x8, 0xc000000000, 0x0, 0x3d10f00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/range_lookup.go:198 +0x8c9 github.com/cockroachdb/cockroach/pkg/storage.(*replicaGCQueue).process(0xc000327480, 0x489e160, 0xc005942660, 0xc00669c800, 0x0, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_gc_queue.go:203 +0xc4 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc005942660, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc005942660, 0xc0002dd200, 0x23, 0xdf8475800, 0xc004608600, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc000159e00, 0x489e1a0, 0xc0046085d0, 0x4906de0, 0xc00669c800, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc004608570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e1a0, 0xc004608570, 0xc0002dd0e0, 0x2d, 0x0, 0x0, 0xc0046085a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6963 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003e36c80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003bfff80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00419f500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 7202 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00071e340, 0xc003439800, 0xc002a437a0, 0xf, 0xc0026c67e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc003055cb0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00424fe20, 0xc0011a2480, 0xc001a3d310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242720 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001290700, 0xc006787c20, 0x489e1a0, 0xc0070f8ab0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 5722 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0027c8f80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 7342 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003607900, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001f90cc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003174780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 7343 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003174780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 242647 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbc, 0x490bae0, 0xc004f66180, 0x0, 0x1, 0xc0012cda00, 0x2e) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbc, 0xc0050518c0, 0xc0026dc800, 0x15c1014bd4ace27f) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc002a0c5c0, 0x33, 0x0, 0x0, 0xc002e5dbf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7337 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc002935ef0, 0xc0031eea50, 0x5, 0x5, 0xc006257e00, 0x1f0, 0xc00784a400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002935ef0, 0xc0031eea50, 0x5, 0x5, 0xc11b10, 0xc003fab860, 0xc0038c7af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003423590, 0xc0031eea50, 0x5, 0x5, 0x76, 0xc0038c7b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003423590, 0xc0031eea50, 0x5, 0x5, 0x5, 0xc0038c7bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc004b8cf00, 0xc0031eea50, 0x5, 0x5, 0x203001, 0x203001, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0031eea40, 0x7fffffff, 0x30, 0xc007a7ef00, 0x4, 0xc0038c7c70, 0xc000b00000, 0x7f086d4ae238) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0031eea40, 0xc004b8cf00, 0x0, 0x0, 0x7fffffff, 0xc0038c7e30, 0x4886820, 0x72b56a8, 0x8, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0031eea40, 0x7f086d36adf0, 0xc000284fd0, 0xc004b8cf00, 0x0, 0x0, 0x3de86e0, 0xc006320e60, 0x7fffffff, 0xc0038c7e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00206ca80, 0x3de86e0, 0xc006320e60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc0028b1c50, 0x489e1a0, 0xc0034236b0, 0xc00784a5f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc0028b1c50, 0xc000214960, 0x489e1a0, 0xc0034236b0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0034236b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0028b1c70, 0xc0011a2480, 0xc003423680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 8611 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc002a6cfa0, 0xc00191ae10, 0x5, 0x5, 0x0, 0x0, 0xc0075ad400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002a6cfa0, 0xc00191ae10, 0x5, 0x5, 0xc11b10, 0xc0003e64e0, 0xc0044a3af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001852ae0, 0xc00191ae10, 0x5, 0x5, 0x3c, 0xc0044a3b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001852ae0, 0xc00191ae10, 0x5, 0x5, 0x5, 0xc0044a3bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003944900, 0xc00191ae10, 0x5, 0x5, 0x203001, 0x203001, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00191ae00, 0x7fffffff, 0x1311b3c80f16c7fa, 0x1311b3c80f16c7fa, 0xc0044a3c78, 0x2c15de0, 0x3a5c620, 0xc000d01020) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00191ae00, 0xc003944900, 0x0, 0x0, 0x7fffffff, 0xc0044a3e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00191ae00, 0x7f086d36adf0, 0xc000284fd0, 0xc003944900, 0x0, 0x0, 0x3de86e0, 0xc007afb960, 0x7fffffff, 0xc0044a3e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00432aa80, 0x3de86e0, 0xc007afb960, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc001a9c480, 0x489e1a0, 0xc001852e10, 0xc0075ad400) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc001a9c480, 0xc0010e50e0, 0x489e1a0, 0xc001852e10, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc001852e10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001a9c4d0, 0xc00108b320, 0xc001852de0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 7344 [IO wait]: internal/poll.runtime_pollWait(0x7f086d4538c0, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc004bc2418, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc004bc2400, 0xc002ff8800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc004bc2400, 0xc002ff8800, 0x75b, 0x75b, 0xc003d83938, 0x701, 0xc00378a880) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0025e4760, 0xc002ff8800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc001466de0, 0xc002ff8800, 0x75b, 0x75b, 0xc002ff880d, 0x5a, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc007ac0940, 0xc002ff8800, 0x75b, 0x75b, 0x77, 0x482c600, 0xc003d83a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00378a958, 0x482c340, 0xc007ac0940, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00378a700, 0x7f086d36ae18, 0xc001466de0, 0x5, 0xc001466de0, 0x5a) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00378a700, 0x40e5100, 0xc00378a838, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00378a700, 0xc004624000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0029116e0, 0xc0013e30d8, 0x9, 0x9, 0xc001684d88, 0xc0072900c0, 0x51) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0029116e0, 0xc0013e30d8, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013e30d8, 0x9, 0x9, 0x482c140, 0xc0029116e0, 0x0, 0x300000000, 0xc0040a8200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013e30a0, 0xc000b0d8f0, 0xc000b0d8f0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003174780, 0xc0011ee5d0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc001a74c00, 0x48fef60, 0xc003174780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc001a74c00, 0x48fef60, 0xc003174780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 10329 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0010e50e0, 0x3, 0xc003ab2540, 0xc00390afc0, 0x48fa520, 0xc003c83330, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003ccae70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003c832b0, 0xc00108b320, 0xc003ccae40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242171 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc003945800, 0xc001897c79, 0xc006e9f5f8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc004700a00, 0x3e99440, 0xc005f7a180, 0xc007ca3650, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc004700a00, 0xc0056f06c0, 0x35) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc002f80900, 0xc007ca3798, 0xc007ca3788, 0xc0074bfef0, 0xc001897c79) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc002f80900, 0x3e99440, 0xc005f7a180, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e160, 0xc005f7a060, 0x3fc5646, 0x21, 0x3f4fce0, 0xc004700900, 0x3e99440, 0xc005f7a180, 0xc004144700, 0xc001697140, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e160, 0xc005f7a060, 0x3fc5646, 0x21, 0x3f4fce0, 0xc004700900, 0x3e99440, 0xc005f7a180, 0xc004144700, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc004144700, 0x489e160, 0xc005f7a060, 0x3fc5646, 0x21, 0x3f4fce0, 0xc004700900, 0x3e99440, 0xc005f7a180, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc002434ad8, 0x489e160, 0xc005f7a060, 0xc004700900, 0x0, 0x0, 0x0, 0x100000002, 0x489e160, 0xc005f7a060) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006f90ac0, 0x489e160, 0xc005f7a060, 0x2, 0x485a560, 0xc002434ad8, 0x0, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006f90ac0, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x200000002, 0x3, 0xc003d4070c, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc0016f4360, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc0016f4360, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc0016f4360, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc0016f4360, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc0016f4360, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc0016f4360, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0015a6c60, 0x489e160, 0xc005f7a060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.lookupRangeFwdScan(0x489e160, 0xc005f7a060, 0x482c760, 0xc0015a6c60, 0xc0018979e0, 0x2, 0x8, 0x0, 0x0, 0x1dcd6500, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/range_lookup.go:328 +0x3f2 github.com/cockroachdb/cockroach/pkg/internal/client.RangeLookup(0x489e160, 0xc005f7a060, 0x482c760, 0xc0015a6c60, 0xc0018979e0, 0x2, 0x8, 0xc000000000, 0x0, 0x3d10f00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/range_lookup.go:198 +0x8c9 github.com/cockroachdb/cockroach/pkg/storage.(*replicaGCQueue).process(0xc002d62660, 0x489e160, 0xc005f7a060, 0xc0028aa000, 0x0, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_gc_queue.go:203 +0xc4 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc005f7a060, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc005f7a060, 0xc0074bfec0, 0x23, 0xdf8475800, 0xc0034dfda0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc000ff3e00, 0x489e1a0, 0xc0034dfce0, 0x4906de0, 0xc0028aa000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc0034dfc80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc0034dfc80, 0xc0074bfe90, 0x2d, 0x0, 0x0, 0xc0034dfcb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 242626 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbc, 0x490bae0, 0xc005622d80, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbc, 0xc002135560, 0xc007610000, 0x15c1014bd3f804ba) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc0042d6400, 0x33, 0x0, 0x0, 0xc000aa7860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6251 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000b61600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 10330 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc004164700, 0xc002134360, 0x489e1a0, 0xc003ccaf30) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 7827 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002c5cb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 6745 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0032be1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 7335 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002935cc0, 0xc0031ee890, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002935cc0, 0xc0031ee890, 0x5, 0x5, 0xc11b10, 0xc003fab840, 0xc004993978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003423350, 0xc0031ee890, 0x5, 0x5, 0x74cff0, 0xc0049939a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003423350, 0xc0031ee890, 0x5, 0x5, 0x5, 0xf, 0xc000079c20, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc004b8ce00, 0xc0031ee890, 0x5, 0x5, 0x83e500, 0xc00558b850, 0xc004993a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0031ee880, 0x7fffffff, 0x3c06120, 0xc000079c20, 0xc004993a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0031ee880, 0xc004b8ce00, 0x0, 0x0, 0x7fffffff, 0xc004993cb8, 0x4886820, 0x72b56a8, 0x0, 0xc005a19660, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0031ee880, 0x7f086d36adf0, 0xc000284fd0, 0xc004b8ce00, 0x0, 0x0, 0x3de8800, 0xc004bd9940, 0x7fffffff, 0xc004993cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc003c12c80, 0x3de8800, 0xc004bd9940, 0xc004993cb8, 0xc007e240f0, 0xc004bd9940) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc003c12c80, 0xc000087470, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc003bcaa20, 0xc004993e00, 0xc004993df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc003bcaa20, 0x3de8800, 0xc004bd9940, 0x2c5e8e0, 0xc000c91500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0028b1b80, 0x489e1a0, 0xc0034233e0, 0xc006de5c40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0028b1b80, 0xc004195380, 0xc0010e50e0, 0x489e1a0, 0xc0034233e0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0034233e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0028b1ba0, 0xc00108b320, 0xc003423380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6782 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453cd0, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0033a4898, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0033a4880, 0xc001f41800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0033a4880, 0xc001f41800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00109ce40, 0xc001f41800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc004d1be20, 0xc001f41800, 0x791, 0x791, 0x6e, 0x482c600, 0xc00177ca78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003911e58, 0x482c340, 0xc004d1be20, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003911c00, 0x48317e0, 0xc00109ce40, 0x5, 0xc00109ce40, 0xc0065f46c0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003911c00, 0x40e5100, 0xc003911d38, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003911c00, 0xc002882000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003c28d80, 0xc0037323b8, 0x9, 0x9, 0xc000abb0d8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003c28d80, 0xc0037323b8, 0x9, 0x9, 0x9, 0xc00777ad20, 0x485e320, 0xc0028b8890) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0037323b8, 0x9, 0x9, 0x482c140, 0xc003c28d80, 0x0, 0xc000000000, 0xc000c9dc00, 0x65f4600) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc003732380, 0xc0065f4690, 0xc0065f4690, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0035a01c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 11775 [select]: github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x489e1a0, 0xc00409ad20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:123 +0x436 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0028b0fa0, 0xc0011a2480, 0xc0028b0f90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 9885 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc004854780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 243070 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152b340, 0x489e120, 0xc00010c010, 0xbc, 0x490bae0, 0xc0071be4e0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c3cea0, 0x489e120, 0xc00010c010, 0xbc, 0xc0051407e0, 0xc00179f738, 0xc0051c2f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc002a0d040, 0x33, 0x0, 0x0, 0xc0069fee70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 241583 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbd, 0x490bae0, 0xc00760bf80, 0x0, 0x1, 0xc001466840, 0xfaa) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbd, 0xc003e4e000, 0xc00386e800, 0x15c1014bca3d71a1) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc00491eb00, 0x33, 0x0, 0x0, 0xc005c85ef0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10146 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003cbde00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002320420, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0032be8c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 6831 [select]: github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x489e1a0, 0xc004103110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:123 +0x436 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003acc760, 0xc000cfe090, 0xc003acc750) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 8445 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000b86a50, 0xc000360330, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000b86a50, 0xc000360330, 0x5, 0x5, 0xc11b10, 0xc001898240, 0xc0044c5978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0010f5680, 0xc000360330, 0x5, 0x5, 0x74cff0, 0xc0044c59a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0010f5680, 0xc000360330, 0x5, 0x5, 0x5, 0xf, 0xc000e8fda0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003c7c100, 0xc000360330, 0x5, 0x5, 0x83e500, 0xc005880550, 0xc0044c5a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000360320, 0x7fffffff, 0x3c06120, 0xc000e8fda0, 0xc0044c5a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000360320, 0xc003c7c100, 0x0, 0x0, 0x7fffffff, 0xc0044c5cb8, 0x4886820, 0x72b56a8, 0x0, 0xc004c23a20, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000360320, 0x7f086d36adf0, 0xc000284fd0, 0xc003c7c100, 0x0, 0x0, 0x3de8800, 0xc005d31140, 0x7fffffff, 0xc0044c5cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc004bd4000, 0x3de8800, 0xc005d31140, 0xc0044c5cb8, 0xc004a4cf50, 0xc005d31140) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc004bd4000, 0xc00009c570, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc004672000, 0xc0044c5e00, 0xc0044c5df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc004672000, 0x3de8800, 0xc005d31140, 0x2c5e8e0, 0xc001f42700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0016b4310, 0x489e1a0, 0xc0010f57a0, 0xc006093240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0016b4310, 0xc004195400, 0xc000214960, 0x489e1a0, 0xc0010f57a0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0010f57a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b4360, 0xc0011a2480, 0xc0010f56b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242843 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbc, 0x490bae0, 0xc007693ec0, 0x0, 0x1, 0xc0012cda00, 0x31) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbc, 0xc005437680, 0xc00522f800, 0x15c1014bd6876ee2) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc0046e4c80, 0x33, 0x0, 0x0, 0xc003d8c900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 143485 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).addToPurgatoryLocked.func2(0x489e1a0, 0xc00495e5a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:1086 +0x325 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0040d8930, 0xc0011a2480, 0xc0058610c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242715 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152b340, 0x489e120, 0xc00010c010, 0xbb, 0x490bae0, 0xc005113260, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c3cea0, 0x489e120, 0xc00010c010, 0xbb, 0xc00483efc0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc001171380, 0x33, 0x0, 0x0, 0xc005c698c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 241707 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xbb, 0x490bae0, 0xc0044c2ba0, 0x0, 0x1, 0xc001466840, 0xfaa) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xbb, 0xc0050786c0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc005d91240, 0x33, 0x0, 0x0, 0xc0043e5ec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10185 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453720, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc003cae518, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc003cae500, 0xc000b0f000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc003cae500, 0xc000b0f000, 0x75b, 0x75b, 0xc0034bc938, 0x701, 0x294a5294a5) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000c74150, 0xc000b0f000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0047c5140, 0xc000b0f000, 0x75b, 0x75b, 0x8, 0x1ebd8d257b98, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc001e27f20, 0xc000b0f000, 0x75b, 0x75b, 0x203001, 0x140, 0xc0034bca50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003f5a258, 0x482c340, 0xc001e27f20, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003f5a000, 0x7f086d36ae18, 0xc0047c5140, 0x5, 0xc0047c5140, 0x0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003f5a000, 0x40e5100, 0xc003f5a138, 0xc00473f500) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003f5a000, 0xc002cba000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002ca5b00, 0xc00152a818, 0x9, 0x9, 0xc00147d648, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002ca5b00, 0xc00152a818, 0x9, 0x9, 0x9, 0xdeb8c002e2783701, 0xde00000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00152a818, 0x9, 0x9, 0x482c140, 0xc002ca5b00, 0x0, 0xd700000000, 0xc0068b1e00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00152a7e0, 0xc00147b500, 0xc00147b500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003c2a000, 0xc0024ed980, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0025ed980, 0x48fef60, 0xc003c2a000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0025ed980, 0x48fef60, 0xc003c2a000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 9886 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00071e340, 0xc002232720, 0xc001896400, 0xf, 0xc003314300, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc003aa5020) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0048844c0, 0xc0011a2480, 0xc00352c730) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242063 [select]: github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).lookupRangeDescriptorInternal(0xc001830b40, 0x489e1a0, 0xc0026655f0, 0xc002d873a3, 0x2, 0xd, 0xc006bfbd80, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:392 +0x4ba github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).LookupRangeDescriptorWithEvictionToken(...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:266 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).getDescriptor(0xc0011138c0, 0x489e1a0, 0xc0026655f0, 0xc002d873a3, 0x2, 0xd, 0xc006bfbd80, 0x0, 0x2, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:480 +0x9a github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc0011138c0, 0x489e1a0, 0xc0026655f0, 0x15c1014bd23d4a2d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1431 +0x231 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc0011138c0, 0x489e1a0, 0xc0026655f0, 0x15c1014bd23d4a2d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc0011138c0, 0x489e1a0, 0xc0026655f0, 0x15c1014bd23d4a2d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc00185c660, 0x489e1a0, 0xc0026655f0, 0x15c1014bd23d4a2d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc00185c600, 0x489e1a0, 0xc0026655f0, 0x15c1014bd23d4a2d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e1a0, 0xc0026655f0, 0xc0056c81b0, 0xc002812b00, 0x0, 0xc0056c8880) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc00185c600, 0x489e1a0, 0xc0026655f0, 0xc002812b00, 0x19, 0xc0056c83e8) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).MaybePushTransactions(0xc000c3cea0, 0x489e1a0, 0xc0026655f0, 0xc0056c8880, 0x15c1014bd23b85a4, 0x0, 0x200000002, 0x3, 0xc002d8733c, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:439 +0x76b github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).maybePushIntents(0xc000c3cea0, 0x489e1a0, 0xc0026655f0, 0xc00563a120, 0x1, 0x1, 0x15c1014bd23b85a4, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:341 +0x25d github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).ProcessWriteIntentError(0xc000c3cea0, 0x489e1a0, 0xc0026655f0, 0xc005a2ecd0, 0x490c5e0, 0xc0029d6cc0, 0x15c1014bd23b85a4, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:258 +0x13b github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000c91500, 0x489e1a0, 0xc002665590, 0x15c1014bd23b85a4, 0x0, 0x200000002, 0x3, 0xc002d8733c, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3106 +0x7e7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc00040c140, 0x489e1a0, 0xc002665590, 0x0, 0x0, 0x200000002, 0x3, 0xc002d8733c, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc002665560, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00108b320, 0x489e1a0, 0xc002665560, 0x3f956b7, 0x10, 0xc0056c9710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0018ff400, 0x489e1a0, 0xc002665560, 0xc003250b80, 0xc002665560, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0018ff400, 0x489e1a0, 0xc002665530, 0xc003250b80, 0xc0018ff400, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc002665530, 0x3f4fce0, 0xc003250b80, 0x3f4fce0, 0xc003250b80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc002665530, 0x3f4fce0, 0xc003250b80, 0xc003b1a700, 0xc003b1a720, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc002665530, 0x3f4fce0, 0xc003250b80, 0xc003b1a700, 0xc003b1a720, 0x6ff238, 0x20, 0x3b5a920, 0xc00310e901) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc002665530, 0x3f4fce0, 0xc003250b80, 0xc003b1a700, 0xc003b1a720, 0xc003676bc8, 0x7edeba, 0x3d2f600, 0xc002665530) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc0018ff400, 0x489e1a0, 0xc002665530, 0xc00310e960, 0xc00102e4e0, 0x489e1a0, 0xc002665530, 0xc00501d200, 0x23) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc001a74c00, 0x48fef60, 0xc003174780, 0xc004b2d200, 0xc0013d49c0, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc003174780, 0xc004b2d200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00462c170, 0xc001a74c00, 0x48fef60, 0xc003174780, 0xc004b2d200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 10145 [IO wait]: internal/poll.runtime_pollWait(0x7f086d4537f0, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc004700218, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc004700200, 0xc0015db000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc004700200, 0xc0015db000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000c1a270, 0xc0015db000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc005fce6a0, 0xc0015db000, 0x791, 0x791, 0x42, 0x482c600, 0xc0034bea78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003f8a258, 0x482c340, 0xc005fce6a0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003f8a000, 0x48317e0, 0xc000c1a270, 0x5, 0xc000c1a270, 0xc005c684e0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003f8a000, 0x40e5100, 0xc003f8a138, 0xc0000b0a80) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003f8a000, 0xc003168000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002320360, 0xc00295b1b8, 0x9, 0x9, 0xc003af3318, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002320360, 0xc00295b1b8, 0x9, 0x9, 0x9, 0xc006842540, 0x485e320, 0xc0053a2f80) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00295b1b8, 0x9, 0x9, 0x482c140, 0xc002320360, 0x0, 0xc000000000, 0xc00762d200, 0x5c68400) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00295b180, 0xc005c68480, 0xc005c68480, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0032be8c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 244165 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc007188460, 0xc00718a250, 0x5, 0x5, 0x0, 0xc0068db998, 0xde3d07) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc007188460, 0xc00718a250, 0x5, 0x5, 0xc11b10, 0xc0030ee820, 0xc0068db5c8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0070f8bd0, 0xc00718a250, 0x5, 0x5, 0x28ab5344, 0xc0068db5f0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0070f8bd0, 0xc00718a250, 0x5, 0x5, 0x5, 0xc007ea0ba9, 0x4, 0x43) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00323d500, 0xc00718a250, 0x5, 0x5, 0x101, 0x58, 0xc0068db6a0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00718a240, 0x7fffffff, 0x72b56a8, 0x3dec160, 0xc0003ee2a0, 0xc0068db748, 0x7ee108, 0xc00669e7e0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00718a240, 0xc00323d500, 0x0, 0x0, 0x7fffffff, 0xc0068db908, 0x4886820, 0x72b56a8, 0x3dec160, 0xc0003ee2a0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00718a240, 0x7f086d36adf0, 0xc000284fd0, 0xc00323d500, 0x0, 0x0, 0x3de8b60, 0xc00718a7a0, 0x7fffffff, 0xc0068db908, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc003065c00, 0x3de8b60, 0xc00718a7a0, 0xc0068db908, 0x1, 0xc00718a7a0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc003065c00, 0x40e, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc006787c20, 0xc0068dba50, 0xc0068dba40, 0x86, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc006787c20, 0x3de8b60, 0xc00718a7a0, 0xc0068dbf10, 0x4) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftSnapshotClient).Recv(0xc0062a9290, 0xc00669e9f0, 0x1, 0xc000000001) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:674 +0x62 github.com/cockroachdb/cockroach/pkg/storage.sendSnapshot(0x489e1a0, 0xc00669e9f0, 0xc000c91538, 0xc000d05500, 0x7f08666bd108, 0xc0062a9290, 0x482f3e0, 0xc0011b8c00, 0x23, 0x11, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:1042 +0xe84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendSnapshot(0xc0010e50e0, 0x489e1a0, 0xc00669e9f0, 0xc000c91538, 0xc0011b8c00, 0x23, 0x11, 0xc004a76cb0, 0xc007188370, 0xc005fb4720, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:681 +0x27b github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendSnapshot(0xc007611800, 0x489e1a0, 0xc00669e9f0, 0x300000003, 0xc000000003, 0xc005505af4, 0x200000001, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1703 +0x5e1 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).atomicReplicationChange(0xc007611800, 0x489e1a0, 0xc00669e9f0, 0xc00053fd50, 0x2, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005505a90, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1151 +0x1f9 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).changeReplicasImpl(0xc007611800, 0x489e1a0, 0xc00669e9f0, 0xc0033c90f8, 0xc000000002, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005505a90, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:966 +0x4eb github.com/cockroachdb/cockroach/pkg/storage.(*Replica).ChangeReplicas(0xc007611800, 0x489e1a0, 0xc00669e9f0, 0xc0033c90f8, 0x2, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005505a90, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:902 +0x224 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc007611800, 0x489e1a0, 0xc00669e9f0, 0xc0079f0c00, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1324 +0x3c8 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc007611800, 0x489e1a0, 0xc00669e9c0, 0xbb, 0x15c1014bde78dedd, 0x0, 0x200000002, 0x2, 0xc004e62c30, 0xbb, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc007611800, 0x489e1a0, 0xc00669e9c0, 0x15c1014bde78dedd, 0x0, 0x200000002, 0x2, 0xc004e62c30, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000c91500, 0x489e1a0, 0xc00669e960, 0x15c1014bde78dedd, 0x0, 0x200000002, 0x2, 0xc004e62c30, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc00040c140, 0x489e1a0, 0xc00669e960, 0x0, 0x0, 0x200000002, 0x2, 0xc004e62c30, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc00669e960, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00108b320, 0x489e1a0, 0xc00669e960, 0x3f956b7, 0x10, 0xc006d5ffa0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0018ff400, 0x489e1a0, 0xc00669e960, 0xc0079f0b80, 0xc00669e960, 0xc00669e720, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0018ff400, 0x489e1a0, 0xc00669e930, 0xc0079f0b80, 0xc00669e810, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:540 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006c86740, 0x489e1a0, 0xc00669e930, 0x2, 0x485fe20, 0xc0013f8090, 0x0, 0x0, 0x200000002, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006c86740, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x200000002, 0x2, 0xc004e62c30, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc0011138c0, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc0011138c0, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc0011138c0, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc0011138c0, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc0011138c0, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc0011138c0, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc00185c660, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc00185c600, 0x489e1a0, 0xc00669e810, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e1a0, 0xc00669e810, 0xc006d61ba0, 0xc003de2100, 0xc0033c90a0, 0xc005504688) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc00185c600, 0x489e1a0, 0xc00669e810, 0xc003de2100, 0xc005504688, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminChangeReplicas(0xc00185c600, 0x489e1a0, 0xc00669e810, 0x3d0b580, 0xc007a910c0, 0xbb, 0xc005504688, 0x3, 0x8, 0xc005504690, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:582 +0x268 github.com/cockroachdb/cockroach/pkg/storage.(*Store).AdminRelocateRange(0xc000c91500, 0x489e1a0, 0xc00669e810, 0xbb, 0xc005504688, 0x3, 0x8, 0xc005504690, 0x2, 0x8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1984 +0x1628 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc007611800, 0x489e1a0, 0xc00669e810, 0xc0079f0a00, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1335 +0x803 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc007611800, 0x489e1a0, 0xc00669e7e0, 0xbb, 0x15c1014bde773d25, 0x0, 0x200000002, 0x2, 0xc0055056ec, 0xbb, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc007611800, 0x489e1a0, 0xc00669e7e0, 0x15c1014bde773d25, 0x0, 0x200000002, 0x2, 0xc0055056ec, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000c91500, 0x489e1a0, 0xc00669e780, 0x15c1014bde773d25, 0x0, 0x200000002, 0x2, 0xc0055056ec, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc00040c140, 0x489e1a0, 0xc00669e780, 0x0, 0x0, 0x200000002, 0x2, 0xc0055056ec, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc00669e750, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00108b320, 0x489e1a0, 0xc00669e750, 0x3f956b7, 0x10, 0xc006d63710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0018ff400, 0x489e1a0, 0xc00669e750, 0xc0079f0980, 0xc00669e750, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0018ff400, 0x489e1a0, 0xc00669e720, 0xc0079f0980, 0xc0018ff400, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc00669e720, 0x3f4fce0, 0xc0079f0980, 0x3f4fce0, 0xc0079f0980, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc00669e720, 0x3f4fce0, 0xc0079f0980, 0xc007a90fa0, 0xc007a90fc0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc00669e720, 0x3f4fce0, 0xc0079f0980, 0xc007a90fa0, 0xc007a90fc0, 0x6ff238, 0x20, 0x3b5a920, 0xc006c88301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc00669e720, 0x3f4fce0, 0xc0079f0980, 0xc007a90fa0, 0xc007a90fc0, 0xc003deabc8, 0x7edeba, 0x3d2f600, 0xc00669e720) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc0018ff400, 0x489e1a0, 0xc00669e720, 0xc006c88300, 0xc00102e4e0, 0x489e1a0, 0xc00669e720, 0xc005070c00, 0x25) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc007ab4600, 0xc0013d49c0, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc007ab4600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003b98d0, 0xc001a74c00, 0x48fef60, 0xc0024ca480, 0xc007ab4600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 59314 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).addToPurgatoryLocked.func2(0x489e1a0, 0xc004298f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:1086 +0x325 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00424fa00, 0xc000cfe090, 0xc00413b680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242678 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152b340, 0x489e120, 0xc00010c010, 0xbb, 0x490bae0, 0xc001fa21e0, 0x0, 0x1, 0x3e8e720, 0x6) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c3cea0, 0x489e120, 0xc00010c010, 0xbb, 0xc004bced80, 0xc00676a000, 0x15c1014bd3f905f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00108b320, 0x489e120, 0xc00010c010, 0xc003a09380, 0x33, 0x0, 0x0, 0xc00184f3b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 8242 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc003632690, 0xc000326850, 0x5, 0x5, 0x0, 0x0, 0xc007ae6600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003632690, 0xc000326850, 0x5, 0x5, 0xc11b10, 0xc00149e8c0, 0xc0036d1af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0006839b0, 0xc000326850, 0x5, 0x5, 0x3a, 0xc0036d1b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0006839b0, 0xc000326850, 0x5, 0x5, 0x5, 0xc0036d1bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0040a8200, 0xc000326850, 0x5, 0x5, 0x203001, 0x203001, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000326840, 0x7fffffff, 0x732a2f, 0xc0018e4900, 0x4, 0xc0036d1c70, 0x733ecb, 0xc001f90360) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000326840, 0xc0040a8200, 0x0, 0x0, 0x7fffffff, 0xc0036d1e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000326840, 0x7f086d36adf0, 0xc000284fd0, 0xc0040a8200, 0x0, 0x0, 0x3de86e0, 0xc007ac09c0, 0x7fffffff, 0xc0036d1e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0030da480, 0x3de86e0, 0xc007ac09c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc00026c720, 0x489e1a0, 0xc000683aa0, 0xc007ae6600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc00026c720, 0xc0010e50e0, 0x489e1a0, 0xc000683aa0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc000683aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c750, 0xc00108b320, 0xc000683a70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 8239 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc004144700, 0xc004c5c7e0, 0x489e1a0, 0xc0006832c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 8241 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0010e50e0, 0x48fa580, 0xc00026c720, 0x485c120, 0xc0010e50e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc0030da480, 0x0, 0xc002911620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc0030da480, 0xc000326880, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0010e50e0, 0x48eb940, 0xc0030da480, 0xc000326880, 0x40e1ba8, 0x489e1a0, 0xc000683a10) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc001a74c00, 0x48fef60, 0xc003174780, 0xc0040a8200, 0xc0013d4330, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc003174780, 0xc0040a8200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00462c170, 0xc001a74c00, 0x48fef60, 0xc003174780, 0xc0040a8200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 8240 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0036324b0, 0xc000326690, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0036324b0, 0xc000326690, 0x5, 0x5, 0xc11b10, 0xc00149e8a0, 0xc004da7978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000683500, 0xc000326690, 0x5, 0x5, 0x74cff0, 0xc004da79a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc000683500, 0xc000326690, 0x5, 0x5, 0x5, 0xf, 0xc000e8fda0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0040a8100, 0xc000326690, 0x5, 0x5, 0x83e500, 0xc005db41b0, 0xc004da7a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000326680, 0x7fffffff, 0x3c06120, 0xc000e8fda0, 0xc004da7a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000326680, 0xc0040a8100, 0x0, 0x0, 0x7fffffff, 0xc004da7cb8, 0x4886820, 0x72b56a8, 0x0, 0xc003239d00, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000326680, 0x7f086d36adf0, 0xc000284fd0, 0xc0040a8100, 0x0, 0x0, 0x3de8800, 0xc00705c040, 0x7fffffff, 0xc004da7cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001196e80, 0x3de8800, 0xc00705c040, 0xc004da7cb8, 0xc005ba83c0, 0xc00705c040) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001196e80, 0xc00009c570, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc004c5c7e0, 0xc004da7e00, 0xc004da7df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc004c5c7e0, 0x3de8800, 0xc00705c040, 0x2c5e8e0, 0xc001f42700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc00026c4e0, 0x489e1a0, 0xc000683590, 0xc000ae10c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc00026c4e0, 0xc001995300, 0xc000214960, 0x489e1a0, 0xc000683590, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc000683590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00026c520, 0xc0011a2480, 0xc000683530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 244164 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00323d000, 0xc0053a3a29, 0xc004c6d298) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc003065300, 0x3e99440, 0xc005112fc0, 0xc0066152f0, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc003065300, 0xc001171180, 0x37) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0067867e0, 0xc006615438, 0xc006615428, 0xc006821530, 0xc0053a3a29) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0067867e0, 0x3e99440, 0xc005112fc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e160, 0xc007775e60, 0x3fc5646, 0x21, 0x3f4fce0, 0xc003065280, 0x3e99440, 0xc005112fc0, 0xc0011ac380, 0xc000ffe980, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e160, 0xc007775e60, 0x3fc5646, 0x21, 0x3f4fce0, 0xc003065280, 0x3e99440, 0xc005112fc0, 0xc0011ac380, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc0011ac380, 0x489e160, 0xc007775e60, 0x3fc5646, 0x21, 0x3f4fce0, 0xc003065280, 0x3e99440, 0xc005112fc0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc00043cfd8, 0x489e160, 0xc007775e60, 0xc003065280, 0x0, 0x0, 0x0, 0x2, 0x489e160, 0xc007775e60) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006411240, 0x489e160, 0xc007775e60, 0x2, 0x485a560, 0xc00043cfd8, 0x0, 0x0, 0x200000002, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006411240, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x200000002, 0x2, 0xc0028168c8, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc0016f4360, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc0016f4360, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0xbb, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc0016f4360, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc0016f4360, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc0016f4360, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc0016f4360, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0015a6c60, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0015a6c00, 0x489e160, 0xc007775e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e160, 0xc007775e60, 0xc004c6f3a8, 0xc004390100, 0x8, 0xc0053a3848) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0015a6c00, 0x489e160, 0xc007775e60, 0xc004390100, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminRelocateRange(0xc0015a6c00, 0x489e160, 0xc007775e60, 0x3cd5280, 0xc005fcf2c0, 0xc0053a3970, 0x1, 0x1, 0xc006322620, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:605 +0xa5 github.com/cockroachdb/cockroach/pkg/storage.(*mergeQueue).process(0xc002d62540, 0x489e160, 0xc007775e60, 0xc005c0d800, 0xc004e90a50, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/merge_queue.go:341 +0xda3 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc007775e60, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc007775e60, 0xc003ea32a0, 0x1f, 0xdf8475800, 0xc006867fb0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc000ff2c80, 0x489e1a0, 0xc006867f80, 0x4906de0, 0xc005c0d800, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc006867f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc006867f20, 0xc006a75200, 0x29, 0x0, 0x0, 0xc006867f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7408 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc000214960, 0x2, 0xc0034397a0, 0xc001995300, 0x48fa520, 0xc00026c4e0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc000683200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00038c680, 0xc0011a2480, 0xc0014381e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 241771 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbd, 0x490bae0, 0xc00533a060, 0x0, 0x1, 0x7, 0xc00122f200) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbd, 0xc0026a2b40, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc004baad00, 0x33, 0x0, 0x0, 0xc00512c780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 241942 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xbd, 0x490bae0, 0xc001466000, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xbd, 0xc0058d8360, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0037f41c0, 0x33, 0x0, 0x0, 0xc005148720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10322 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc00109fb30, 0xc003197d10, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00109fb30, 0xc003197d10, 0x5, 0x5, 0xc11b10, 0xc003f47720, 0xc003715af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003d99440, 0xc003197d10, 0x5, 0x5, 0x0, 0xc003715b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003d99440, 0xc003197d10, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00288bb00, 0xc003197d10, 0x5, 0x5, 0x0, 0x300000003, 0x3) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003197d00, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003197d00, 0xc00288bb00, 0x0, 0x0, 0x7fffffff, 0xc003715e30, 0x4886820, 0x72b56a8, 0x8, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003197d00, 0x7f086d36adf0, 0xc000284fd0, 0xc00288bb00, 0x0, 0x0, 0x3de86e0, 0xc00637e840, 0x7fffffff, 0xc003715e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0047fd500, 0x3de86e0, 0xc00637e840, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc003a19070, 0x489e1a0, 0xc003d99500, 0xc003fb41f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc003a19070, 0xc000214960, 0x489e1a0, 0xc003d99500, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003d99500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003a19090, 0xc0011a2480, 0xc003d994d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 7341 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003607940, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0029119e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000359dc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 7340 [IO wait]: internal/poll.runtime_pollWait(0x7f086d453990, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0037b0118, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0037b0100, 0xc002136800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0037b0100, 0xc002136800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000c745f0, 0xc002136800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc006e0d920, 0xc002136800, 0x791, 0x791, 0x7c, 0x482c600, 0xc003d87a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003488958, 0x482c340, 0xc006e0d920, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003488700, 0x48317e0, 0xc000c745f0, 0x5, 0xc000c745f0, 0xc006cdaf90) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003488700, 0x40e5100, 0xc003488838, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003488700, 0xc0036fe000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002911800, 0xc0013e31b8, 0x9, 0x9, 0xc006500418, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002911800, 0xc0013e31b8, 0x9, 0x9, 0x9, 0xc0048ded80, 0x485e320, 0xc006f0eb50) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0013e31b8, 0x9, 0x9, 0x482c140, 0xc002911800, 0x0, 0xc000000000, 0xc0064de200, 0x6cdaf00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0013e3180, 0xc006cdaf60, 0xc006cdaf60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000359dc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 10321 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc000214960, 0x48fa580, 0xc003a19070, 0x485c120, 0xc000214960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc0047fd500, 0x0, 0xc003c28c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc0047fd500, 0xc003197d20, 0x40e1ba8, 0x3c89ac0, 0xc000107e01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc000214960, 0x48eb940, 0xc0047fd500, 0xc003197d20, 0x40e1ba8, 0x489e1a0, 0xc003d994a0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc0025ed980, 0x48fef60, 0xc00419f500, 0xc00288bb00, 0xc000c792f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0025ed980, 0x48fef60, 0xc00419f500, 0xc00288bb00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc004547330, 0xc0025ed980, 0x48fef60, 0xc00419f500, 0xc00288bb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 135473 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x489e1a0, 0xc0010ad050) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:258 +0x11f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e1a0, 0xc0010ad050, 0xc00120de60, 0x2a, 0x0, 0x0, 0xc003d972f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 52705 [select]: github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x489e1a0, 0xc004b22810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:123 +0x436 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00363d050, 0xc00108b320, 0xc00363d040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 240831 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc0002f2000, 0x489e120, 0xc00010c010, 0xbc, 0x490bae0, 0xc006b86e40, 0x0, 0x1, 0x0, 0xc0044e01e0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000a9d790, 0x489e120, 0xc00010c010, 0xbc, 0xc002f750e0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000cfe090, 0x489e120, 0xc00010c010, 0xc004860980, 0x33, 0x0, 0x0, 0xc0060555c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 241589 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xbe, 0x490bae0, 0xc003746f00, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xbe, 0xc00551ad80, 0xc007610000, 0x15c1014bc287e92e) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc005981900, 0x33, 0x0, 0x0, 0xc006a64480) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 241418 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00152a2a0, 0x489e120, 0xc00010c010, 0xbb, 0x490bae0, 0xc003e2de60, 0x0, 0x1, 0x7, 0xc001541b90) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000c769c0, 0x489e120, 0xc00010c010, 0xbb, 0xc00308d200, 0xc0042cfb00, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0011a2480, 0x489e120, 0xc00010c010, 0xc0017d9f00, 0x33, 0x0, 0x0, 0xc0049bdf50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 243098 [select]: github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*contentionQueue).add(0xc0002f1ec0, 0x489e1a0, 0xc004fd7ec0, 0xc005ee0cc0, 0x15c1014bd9f1e51e, 0x0, 0x200000002, 0x3, 0xc0067d9340, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/contention_queue.go:214 +0x853 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).ProcessWriteIntentError(0xc000c3cea0, 0x489e1a0, 0xc004fd7ec0, 0xc006ea72c0, 0x490c5e0, 0xc006de4280, 0x15c1014bd9f1e51e, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:252 +0x2ff github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000c91500, 0x489e1a0, 0xc004fd7e60, 0x15c1014bd9f1e51e, 0x0, 0x200000002, 0x3, 0xc0067d9340, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3106 +0x7e7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc00040c140, 0x489e1a0, 0xc004fd7e60, 0x0, 0x0, 0x200000002, 0x3, 0xc0067d9340, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc004fd7e30, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00108b320, 0x489e1a0, 0xc004fd7e30, 0x3f956b7, 0x10, 0xc0068a9710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0018ff400, 0x489e1a0, 0xc004fd7e30, 0xc005abd280, 0xc004fd7e30, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0018ff400, 0x489e1a0, 0xc004fd7e00, 0xc005abd280, 0xc0018ff400, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc004fd7e00, 0x3f4fce0, 0xc005abd280, 0x3f4fce0, 0xc005abd280, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc004fd7e00, 0x3f4fce0, 0xc005abd280, 0xc005ee0c40, 0xc005ee0c60, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc004fd7e00, 0x3f4fce0, 0xc005abd280, 0xc005ee0c40, 0xc005ee0c60, 0x6ff238, 0x20, 0x3b5a920, 0xc005d61501) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc004fd7e00, 0x3f4fce0, 0xc005abd280, 0xc005ee0c40, 0xc005ee0c60, 0xc004274bc8, 0x7edeba, 0x3d2f600, 0xc004fd7e00) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc0018ff400, 0x489e1a0, 0xc004fd7e00, 0xc005d61500, 0xc00102e4e0, 0x489e1a0, 0xc004fd7e00, 0xc007020600, 0x24) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc001a74c00, 0x48fef60, 0xc0018d2c00, 0xc006cd1500, 0xc0013d49c0, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc001a74c00, 0x48fef60, 0xc0018d2c00, 0xc006cd1500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002b2f10, 0xc001a74c00, 0x48fef60, 0xc0018d2c00, 0xc006cd1500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 244792 [runnable]: github.com/cockroachdb/cockroach/pkg/storage/engine._Cfunc_MVCCScan(0x7f083d81c3c0, 0xc003041324, 0x4, 0xc003041328, 0x1, 0x15c1014be8954cd2, 0x0, 0x9, 0x0, 0x0, ...) _cgo_gotypes.go:1364 +0x6a github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBIterator).MVCCScan.func1(0xc005228078, 0xc003041324, 0x4, 0x4, 0xc003041328, 0x1, 0x1, 0x15c1014be8954cd2, 0x0, 0x9, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2450 +0x23f github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBIterator).MVCCScan(0xc005228078, 0xc003041324, 0x4, 0x4, 0xc003041328, 0x1, 0x1, 0x9, 0x15c1014be8954cd2, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2445 +0x130 github.com/cockroachdb/cockroach/pkg/storage/engine.mvccScanToKvs(0x489e1a0, 0xc00759ff80, 0x4928240, 0xc005228078, 0xc003041324, 0x4, 0x4, 0xc003041328, 0x1, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/mvcc.go:2143 +0xcc github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCScan(0x489e1a0, 0xc00759ff80, 0x48eaaa0, 0xc005228000, 0xc003041324, 0x4, 0x4, 0xc003041328, 0x1, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/mvcc.go:2252 +0x221 github.com/cockroachdb/cockroach/pkg/storage/batcheval.Scan(0x489e1a0, 0xc00759ff80, 0x4920740, 0xc005228000, 0x4933780, 0xc001fbf800, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/batcheval/cmd_scan.go:59 +0x79c github.com/cockroachdb/cockroach/pkg/storage.evaluateCommand(0x489e1a0, 0xc00759ff80, 0x0, 0x0, 0x0, 0x4920740, 0xc005228000, 0x4933780, 0xc001fbf800, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_evaluate.go:416 +0x235 github.com/cockroachdb/cockroach/pkg/storage.evaluateBatch(0x489e1a0, 0xc00759ff80, 0x0, 0x0, 0x4920740, 0xc005228000, 0x4933780, 0xc001fbf800, 0x0, 0xc00007d500, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_evaluate.go:262 +0x335 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeReadOnlyBatch(0xc001fbf800, 0x489e1a0, 0xc00759ff80, 0xc00007d500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_read.go:94 +0x552 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc001fbf800, 0x489e1a0, 0xc00759ff50, 0x1, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, 0xc003a8a3b4, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:578 +0x2c9 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc001fbf800, 0x489e1a0, 0xc00759ff50, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, 0xc003a8a3b4, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000c91500, 0x489e1a0, 0xc00759fef0, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, 0xc003a8a3b4, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc00040c140, 0x489e1a0, 0xc00759fef0, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, 0xc003a8a3b4, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc00759fef0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00108b320, 0x489e1a0, 0xc00759fef0, 0x3f956b7, 0x10, 0xc005863d50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0018ff400, 0x489e1a0, 0xc00759fef0, 0xc00007d480, 0xc00759fef0, 0xc002665530, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0018ff400, 0x489e1a0, 0xc00759fec0, 0xc00007d480, 0xc00759fe60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:540 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc00731a280, 0x489e1a0, 0xc00759fec0, 0x2, 0x485fe20, 0xc0013f8090, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc00731a280, 0x489e1a0, 0xc00759fe60, 0x15c1014be8954cd2, 0x0, 0x200000002, 0x3, 0xc003a8a3b4, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0x15c1014be8954cd2, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0x15c1014be8954cd2, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0x15c1014be8954cd2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0x15c1014be8954cd2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0x15c1014be8954cd2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.lookupRangeFwdScan(0x489e1a0, 0xc00759fe60, 0x482c860, 0xc0011138c0, 0xc002d873a3, 0x2, 0xd, 0x1, 0x8, 0x1dcd6500, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/range_lookup.go:328 +0x3f2 github.com/cockroachdb/cockroach/pkg/internal/client.RangeLookup(0x489e1a0, 0xc00759fe60, 0x482c860, 0xc0011138c0, 0xc002d873a3, 0x2, 0xd, 0x1, 0x8, 0xd35100, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/range_lookup.go:198 +0x8c9 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).RangeLookup(0xc0011138c0, 0x489e1a0, 0xc00759fe60, 0xc002d873a3, 0x2, 0xd, 0x489e100, 0xc00759fe60, 0xc005e1fb00, 0xc003445900, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:359 +0xd5 github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).performRangeLookup(0xc001830b40, 0x489e1a0, 0xc00759fe60, 0xc002d873a3, 0x2, 0xd, 0x4913e00, 0xc0003ee2a0, 0x3, 0xc002d501a8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:449 +0x2ca github.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).lookupRangeDescriptorInternal.func3(0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:322 +0x131 github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc001830b78, 0xc006e6a050, 0xc003041310, 0xd, 0xc00731a180) /go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128 +0x2e created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan /go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:121 +0x29d **************************************************************************** This node experienced a fatal error (printed above), and as a result the process is terminating. Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a problem in CockroachDB. With your help, the support team at Cockroach Labs will try to determine the root cause, recommend next steps, and we can improve CockroachDB based on your report. Please submit a crash report by following the instructions here: https://github.com/cockroachdb/cockroach/issues/new/choose If you would rather not post publicly, please contact us directly at: support@cockroachlabs.com The Cockroach Labs team appreciates your feedback. I190903 18:32:50.779986 1174 sql/event_log.go:130 [n1,client=127.0.0.1:39262,user=root] Event: "create_table", target: 96, info: {TableName:data.public."single col range partitioning" Statement:CREATE TABLE "single col range partitioning" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root} ERROR: exit status 255 context canceled Error: exit status 1 648 runs completed, 0 failures, over 13m30s context canceled FAIL Makefile:973: recipe for target 'roachprod-stress' failed make: *** [roachprod-stress] Error 1